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Abstract. We obtain necessary and sufficient conditions for tlie admissible vectors 
of a new unitary non irreducible representation U. The group G is an arbitrary 
semidirect product whose normal factor A is abelian and whose homogeneous factor 
i? is a locally compact second countable group acting on a Riemannian manifold 
M. The key ingredient in the construction of [/ is a intertwining map between 
the actions of H on the dual group A and on AI. The representation U generalizes 
the restriction of the mctaplcctic representation to triangular subgroups of S'p(d, M), 
whence the name "mock mctaplcctic" . For simplicity, we content ourselves with the 
case where A = R" and AI = M.'^. The main technical point is the decomposition of 
U as direct integral of its irreducible components. This theory is motivated by some 
recent developments in signal analysis, notably shearlets. Many related examples are 
discussed. 



Unitary representations of semidirect products have been thoroughly studied by 
many authors and are useful in a wide variety of applications. In particular, they 
play a central role in the harmonic analysis of the continuous wavelet transform, as 
discussed in [18]. From the point of view of applications, a unitary representation U 
of a locally compact group G (with Haar measure dg) is particularly useful if it yields 
a reproducing formula, that is, a weak reconstruction of the form 



valid for every / in the representation space "H, for some admissible vector rj G T-L. In 
this case [G, U, rj) is called a reproducing system. Alternatively, we simply say that 
G is a reproducing group. If U is irreducible, this is nothing else but the classical 
concept of square integrable representation [12], [13] • Typically, 1-i = L^(M'^), and in 
this case an admissible vector t] is sometimes called a generating function or wavelet. 
Apart from direct use, formula ([T]) is important also because it is the starting point 
for its discrete counterparts, an aspect that we shall not develop in the present paper. 
It is actually rather interesting to observe that most formulae of the above type that 
appear in applications, either in their continuous or discrete versions, turn out to be 
expressible by taking the restriction of the metaplectic representation to some parabolic 
subgroup G of the symplectic group 5*^(6?, M). This is the main theme in the papers 
[9]; [Z]; [H] and the present contribution is an outgrowth thereof. 
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We will be concerned with groups G that are semidirect products, where the normal 
factor is an abelian group A and the homogeneous factor is a locally compact second 
countable group H . Our main object of study is a unitary representation U oi G 
whose construction is based on the following ingredients: a Riemannian manifold M 
on which H acts by diffeomorphisms and a map $ : M — )■ A (the dual group 
of A) that intertwines the actions of if on M and on A. The representation g Ug 
acts on L'^{M) as pointwise multiplication by the character ($(■), (?) if (7 G A and quasi 
regularly ii g E H , as clarified below in f lTU]) . For simplicity, we take A = M" and 
M = M"^ and we also suppose that the Jacobian of the action on is constant. We 
call U the "mock" metaplectic representation because its definition is inspired by the 
case where is a vector space of dxd symmetric matrices on which a closed subgroup 
H of GL{d,M.) acts by cr H- ^h~^ah~^. Under these circumstances, G can be identified 
with a triangular subgroup of Sp{d, M) and U is the restriction to G of the metaplectic 
representation (see Example 1). 

General admissibility criteria for type-I groups have been proved in [18]. Given 
the representation U on Ti, his theory stems from knowledge of a direct integral de- 
composition U = jQim^a dv{a) into irreducible components, and the corresponding 
decomposition "H = jQmfj'Hadv{a). With these data at hand, Fiihr proves that if G 
is non-unimodular, then ([1]) holds true for some 77 if and only if v has density with 
respect to yUg, the Plancherel measure of G; if G is unimodular, then one has to add 
the extra conditions that m„ < dimT/o- for i/-almost every a and jQm„dv{a) < +00. 
Observe that the measure v is known to exist [12], but one has to find it, together 
with the measurable field {1-ia} and the multiplicity function a (-> mg.. The explicit 
knowledge of /ig is also non trivial, in general, but is understood for semidirect prod- 
ucts [23]. Without using the remarkable machinery of [IB], we explicitly decompose U 
and thereby obtain, as a byproduct, computable admissibility criteria in terms of the 
intertwining map $. 

Our finer results are Theorem |8] and Theorem [9l which deal with the cases where 
G is unimodular or non-unimodular, respectively. They both hold under the standard 
technical assumption that the if-orbits are locally closed in $(M'^) and assuming also 
that almost all if-stabilizers in $(M'^) are compact. The latter assumption may be 
removed and yields the weaker conclusion given in Theorem O Theorem IH] actually 
contains the following result: if G is non-unimodular U is reproducing if and only if 
the set of critical points of $ has Lebesgue measure zero. This is of course very easy 
to check in the examples in which $ is explicitly known. In the case where n = d and 
where $ is a homogeneous polynomial, circumstances that happen in many examples, 
then U is reproducing if and only if G is non unimodular and the stabilizers are almost 
all compact (see Theorem [TO|l . This last result settles the problem that was the original 
motivation of this work. 

Here is an outline of the other results contained in the paper. 

• Theorem [H which establishes an important necessary condition for a reproduc- 
ing formula (JT]) to hold true: $ must map sets of positive measure into sets of 
positive measure, hence the critical points C have zero Lebesgue measure and 
n < d. Thus we introduce an open if-invariant subset X of with negligible 
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Lebesgue complement whose image is denoted hj Y = C M" in such 

a way that $ is a submersion of X onto Y. The fibers ^~^{y) are therefore 
Riemannian submanifolds of X and play a crucial role in what follows. All the 
results except Theorem [T] will be formulated for X and Y, namely, for the map 
$ : X — >■ F, and hold true under the assumption that C has zero Lebesgue 
measure (see Assumption 1). 

• Theorem HI based on the classical coarea formula, shows how the Lebesgue 
measure of X disintegrates into a family of measures {I'y} concentrated on 
the fibers ^~^{y), whose covariance with respect to the if-action is explicitly 
calculated in f l22|) . 

• Theorem [3], where a first reduction criterion for admissible vectors is given. 
One looks at the if-orbits in Y and takes their preimages under $ in X. Upon 
selecting an origin y in each if-orbit in Y, one gets the fiber $~^(?/). The 
theorem states that it is necessary and sufficient to test that, for almost every 
iJ-orbit in Y, the L^-norm with respect to Uy of any u G L'^{X,Uy) can be 
reproduced by the (weighted) iJ-integral of the square modulus \{u,riy)^J'^ of 
the components of u along the if-translates of the restriction to $~^(y) of the 
admissible vector rj. This is formula fl2^ . 

• Theorem which exhibits a direct integral decomposition of U in terms of 
induced representations of isotropy subgroups of H, and is independent of any 
admissibility issue. This is achieved as follows. 

— First of all, we assume that the if-orbits are locally closed in Y. This is a 
standard assumption, without which most results in the current literature 
on these themes cannot be applied. In Section [3.4.1l we make some technical 
comments on this in relation to the recent results in [T9] . 

— Secondly, we derive a disintegration of the Lebesgue measure on Y d la 
Mackey, that is, dy = J^r^dX^z). Here A is a pseudo-image measure on 
the locally compact second countable space Z which is a nice parametriza- 
tion of the orbits (better than Y/H) and is concentrated on the orbit 
corresponding to z ^ Z. This preliminary disintegration is carried out in 
Theorem m where the covariance of {r^} with respect to the if-action is 
also calculated in fl28|) . 

— In Proposition [3] we use the measures {r^} in order to "glue" together the 
measures Uy for all y in the same orbit, thereby producing new measures 
fiz = Jy i^y dTziy) on X which, in turn, allow to disintegrate the Lebesgue 
measure on X a.s dx = fizdX{z). As before, the covariance of {/iz} 
with respect to the iJ-action is calculated. The reason for introducing 
these measures are formulae flMl) and fl35|) : the representation space of U, 
namely L^(X), is formally the double direct integral 



Jz JY 

where the inner integral is L^(X, /x^). 
— Next we show in Lemma O that L'^{X,fiz) is unitarily equivalent to the 
representation space Hz of the representation which is unitarily induced 
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to G by the quasi regular representation of the stabilizer -^0(2) (naturally 
extended to the semidirect product R" x Ho(z))- Here it is important to 
select an origin o{z) of the orbit in Y whose label is z. 
The conclusion of Theorem [S] is that U is equivalent to f^WzdX{z), with 
an explicit intertwining isometry. The main technical ingredient of this part 
is the theory of disintegration of measures, as developed by Bourbaki and it 
is reviewed in Appendix lA.ll under the simplifying assumption that the spaces 
are second countable. 
• Theorem [7] assumes that the stabilizers of the H action on Y are almost all 
compact and it is based on the theory of von Neumann algebras. It takes care 
of a nontrivial measurability issue involved in the decomposition of the map 
z I—)- Wz as direct sum of its irreducible components. 

Finally, in Section S] we illustrate several examples. 

2. Notation and assumptions 

In this section we fix the notation and describe the setup. We start by recalling 
the notions of reproducing group and admissible vector. For a thorough discussion the 
reader is referred to [TB] . 

Let G be a locally compact group with (left) Haar measure dg and U he a strongly 
continuous unitary representation of G acting on the complex separable Hilbert space 
"H. A vector 77 G "H is called admissible if 

ll/f = / \{f,Ugv)\'dg ioraW fen. 
Jg 

If such a vector exists, we say that G is a reproducing group and that f/ is a reproducing 
representation. Clearly, if U is reproducing, then it is a cyclic representation, but in 
general it is not irreducible. When U is irreducible, the representation is reproducing 
if and only if it is square integrable [13] . 

2.1. The semidirect product. Let if be a locally compact second countable group 
acting on M" by means of the continuous representation 

(2) y ^ h[y], heH. 

Let G be the semidirect product G = W"' xi H with group law 

(ai, /ii)(a2, = (fli + h\[a2],hih2) ai, 02 e M", hi, /i2 e H, 

where h^ [■] is the action given by the contragredient representation of H on defined 
via the usual inner product by 

(3) {h^[a],y) = {a,h-'[y]), a,yeM^ 

Since h[-] is linear, the semidirect product is well defined and G is a locally compact 
second countable group. Conversely, any locally compact second countable group G 
that is the semidirect product of a closed subgroup and a normal subgroup, which is a 
real vector space of dimension n, is of the above form. 
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The (left) Haar measures of G and H are written dg and dh, and, similarly, da is 
the Lebesgue measure on M". The modular functions of G and H are denoted by Ac 
and A//, respectively. The following relations are easily established 

(4) dg = ——da dh 

a[h) 

(5) ^^C-")-^ 

where a : — (0, +oo) is the character of H defined by 

(6) a{h) = I det(a ^ h^[a])\ = \ det{y ^ h~^[y])\. 
The Fourier transform : L^(]R") — )■ L^(M") is defined by 

= [ e-2-^<^''^V(«) da, f e L\R-) n ^^(R"). 

In general, if G is any locally compact second countable group, L'^iG) will denote the 
Hilbert space of square integrable functions with respect to left Haar measure. Finally, 
if X is a locally compact second countable topological space, the Borel a-algebra on 
X is denoted B{X) and Cc{X) denotes the space of complex continuous functions on 
X with compact support. By measure we mean a a-additive function /i on B{X) with 
values in [0, +oo] which is finite on compact sets. The hypothesis on X implies that 
any such measure is automatically inner and outer regular [26] . A function f : X X' 
between two such spaces will be called Borel measurable if f~^{B) G B{X) for every 
B G B{X') and //-measurable if f~^{B) G B^{X), where -B^(X) denotes the completion 
of B{X) with respect to /i. When dealing with open subsets of Euclidean spaces 
endowed with the Lebesgue measure, however, we say measurable to mean Lebesgue 
measurable. Finally, if G B{M.'^) we write \E\d for it Lebesgue measure or simply \E\ 
if no confusion arises. 

2.2. The mock metaplectic representation. Suppose we are given: 

(HI) a continuous action of H on R'^ by smooth maps denoted x i— ?• h.x, whose 
Jacobian is constant and equal to (3{h); for h & H and E G B{W^) we thus have 

(7) \h.E\=m\E\, 
that is, for every (p G Cc(M'^) 

(f{h~^.x)dx = [5{h) / ip{x)dx. 

(H2) a C^-map $ : R'^ — intertwining the two actions of H on R'^ and R": 

(9) ^{h.x) = h[^{x)] xeR'^,heH. 
For g = {a,h) eG we define Ug : ^^(R'^) ^ ^^(M^) by 

(10) {Ugf){x) = /3(/i)-^e-2'^'<*(")'">/(/i-\a;) 
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for almost every x G M.'^. We show below that this is indeed a representation, that we 
call the mock metaplectic representation. For a motivation for the choice of this name, 
see Example [3] below. 

Remark 1. The representation ([2]) of on plays no direct role in the definition of 
U; its purpose is to construct the semidirect product G. 

Remark 2. Occasionally, we shall write f^{x) for f{h~^.x). 

Remark 3. At this stage there are no limitations on the relative sizes of n and d, but 
we shall see later (Theorem [1]) that in the situations that are of interest to us n < d. 

The next proposition records that flTUl) is a good definition. 

Proposition 1. The map g ^ Ug is a strongly continuous unitary representation of 
G acting on /.^(R'^). 

Proof. Clearly, Ug is a unitary operator and f/ is a representation of M" and H sep- 
arately. In order to prove that it is a representation of G, it is enough to show that 
UhUaUh-i = Uh^a] for a eW and he H. For / e L'^{R'^), and almost every xeR"^ 

{UnUaU^^-.f) (x) = /3(/,)-|e-2-<*(^"^-)''^) {U,,-.f){h-\x) 

To show strong continuity, it is enough to prove that g i-> (f/g/i, /2) is continuous at 
the identity whenever /i,/2 are continuous functions with compact support, and this 
is an easy consequence of the dominated convergence theorem. □ 

2.3. Examples. There are many interesting examples of the setup we are considering. 
We will focus on some situations in which most relevant features occur. 

Example 1. Let if be a closed subgroup of GL{d^R) and assume n = d. Since the 
group H acts naturally on M*^, define 

h.x = h[x] = X X eR'^, h e H. 

Choosing = x, the representation U is equivalent to the quasi regular represen- 
tation of G via the Fourier transform. Necessary and sufficient conditions for U to be 
reproducing are given in [18]. It is worth observing that if n = 1, then H = M+ and 
hence G is the "ax -|- b" group, whereby the dilations are parametrized by H. In this 
case U is 

Uit,a)f{x) = V^e-'^'''^fiax) 
which, after conjugation with the Fourier transform, is the usual wavelet representation. 
It may be generalized to higher dimension [27] . 

Example 2. The Schrodinger representation of the Heisenberg group may be in- 
cluded in this setup, by regarding as a closed subgroup of GL(3,M): 

'l q t 
= \ 1 p 
1 



g,p,tGM}. 
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It is easy to see that is isomorphic to the semidirect product A xi H, where A = 
{[?] • ^ ^} ^i^d H = {[li] '■ q e M}. Indeed, the group H has the natural 
representation on 



q I— )■ 



p2. 



"1 0" 


-1 


"1 -q 


\ 




1 



and acts on M via the translations q.x = x + q. The smooth map $ : 
defined by = [~f ] satisfies the intertwining property 
representation takes the form 



fix) 



-27ri(<J>(x), 



X] 



. The mock metaplectic 
2niit-px) fl^x-q) 



and it thus coincides with the Schrodinger representation, which is irreducible but 
notoriously not square integrable (i.e. not reproducing). Notice that n> d. 

Example 3. This class of examples is where our investigation started. It will be trans- 
parent that the mock metaplectic representation is a generalization of the metaplectic 
representation as restricted to this class of subgroups of Sp{d, M) which includes all the 
parabolic subgroups. Let G = S xi if c Sp{d, M) be a subgroup of the form 



G 



h 
ah 



where if is a closed subgroup of GL{d, M) and S is an n- dimensional subspace of 
Sym((i, M), the space of symmetric dx d matrices. We call any such group a triangular 
subgroup. 

Inner conjugation within G yields the ii-action on S 



(12) 



cr 



under which S must be invariant. As the notation suggests, f jT2|) can be seen as a 
contragredient action. Indeed, we endow Sjm{d, M) with the natural inner product 
(o"i,cr2) = tr(cricr2), whose restriction to S will be denoted (■,■)£• If cr h[a] is the 
representation whose contragredient version is flT^ . then for a, r G S we have 

(r,/i[a])s = Chrh,a)j,=tT{rha'h) = {r,P^{ha%)^, 

where is the orthogonal projection from Sym((i, M) onto S. Thus 

(13) h[a]= P^{h(y'h) aeJ:,heH, 



and if *if = H there is no need of the projection. 

The group H acts naturally on R'^, that is, h.x = hx. Given x G 
be defined by 



let ^(x) G S 



(14) 



tr($(x)o-) 



-- {ax,x) 



X G 



Identifying ~ S S, we can interpret $(x) either as the linear functional on S 
whose action on a is {ax,x) or as the symmetric matrix associated to it via the 
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usual inner product on symmetric matrices. Condition 
observing that a = -Pe(o") and that Ps is self-adjoint, 



is satisfied, since, upon 



tr($(/i.a;)cr) = 
The representation ffTOl) is 



- {^hahx, x) 



ii{^{xYhah) = tr(/i$(x)*/icr) = tr(/i[$(x)]o-). 



(15) 



U{a,h)f{x) = I deth\ 



-1/2 TTi{ax,x) 



f{h- 



X] 



and hence it coincides with the restriction of the metaplectic representation to the 
group G. Various properties of U are analyzed in [THH]. 

An important explicit example in this class is connected to the theory of shearlets 
initiated in [20]. Here the group G parametrizes the two-dimensional phase-space 
operations of translation, dilation and shear and is thus sometimes denoted TDS{2). 
We shall do so and call it the shear let group. 

Precisely, G = M."^ xi H in the following way. Fix a parameter 7 > (usually 
7 = 1/2). The abelian normal subgroup S ~ consists of the 2x2 symmetric 

. The homogeneous group H contains all the 2x2 matrices of the 

e M, t G M+ and 



matrices 



ai 02/2 
02/2 



form SfAf where 



At 



r 











with Haar measure dh = P "^didt and modular function A//(£, t) 
h = [i, t) the linear action on the abelian normal factor 



For any 



e is 



'1 e 




't 0" 


1 




p 



and the group law of G is 

(a, £, t) (a', f, t') = (« + [ *n + tt') . 

It is easy to see that formula f|T^ implies that ^{xi,X2) = —^{xl,xiX2). The mock 
metaplectic representation U restricted to S is equivalent to translations and restricted 
to {At} it amounts to dilations, as shown in [7], where necessary and sufficient condi- 
tions for admissible vectors are given in the case 7 = 1. Admissibility conditions are 
also given in [TU] for 7 = 1/2. Observe that d = n. 

Example 4. This is a case where n < d. Let H = M4. x T. Here T is the one- 
dimensional torus, parametrized by ^ G [0,27r), with Haar measure d6/27r, and M+ is 
the multiplicative group with Haar measure t~^dt where dt is the restriction to M+ 
of the Lebesgue measure on the real line. Hence H has Haar measure dtd6/27rt and 
modular function A//(/i) = 1. The representation of on M is 



where h = {t, 9). Hence in particular a{h) = t~^. The group law in G 
(ai, ti, 9i){a2, t2, 62) = (ai + tj^^Os, ^1^2, Oi + 62). 



X if is 
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The resulting Haar measure is tdtdO /2n and the modular function is easily seen to be 
Ada, t, d)=t^. The action oi h = (t, 9) e H on is given by 

h.{xi,X2) = t(cos9 Xi — sin^ X2, sin6' xi + cos 6'x2) (a^i, 2:2) G 

so that 13(h) = . Finally, $ : — R is given by $(xi,X2) = x\ + x\. The mock 
metaplectic representation Uol G on L^(M^) is 

X / (t~"'^(cos 6 xi + sin 6 X2),t~^{— sin 6 xi + cos6' X2)) . 

Example 5. The point of this example, where again n < d, will become clearer later, 
when iJ-stabilizers enter into the picture: this is a case where they are not compact. 
Let if = R* X M where M* is the (non-connected) multiplicative group of non-zero real 
numbers and M is the additive group with Haar measures and db respectively. 

The Haar measure of H is \t\~^dtdb and = 1. An element h = {t,h) E H acts on M 
and M? by means of 

h[y] =ty y G M 

h.{xi,X2) = {Xi +h,tX2) (Xi,X2)G]R^ 

so that a{h) = \t\~^ and l3{h) = \t\. Finally $ : R^ R is defined by ^{xi,X2) = X2, 
which clearly satisfies (Q. 

3. Main results 

3.1. Dimensional constraints. Our first result. Theorem [H states that if G is re- 
producing, then n < d. The interpretation of this statement in the case of wavelets 
is that the dimension of the space of translations cannot exceed that of the "ground" 
space. In order to prove the theorem we need a technical lemma, in the proof of which 
we use a standard result in harmonic analysis on locally compact abelian groups (see 
Theorem (31.33) in [22]). This is the fact that if a bounded measure u on the lo- 
cally compact abelian group Q has Fourier transform that coincides almost everywhere 
(on the character group Q) with the Fourier transform of an L^(^)-function F, with 
1 < J) < 2, then F G L}{Q), v is absolutely continuous with respect to Haar measure 
and its Radon-Nikodym derivative is F . We apply this to a bounded measure on M". 

Lemma 1. For any f\ri E L^(R'^) the following facts are equivalent: 

(i) JG\{f,Ugv)\^dg < +00; 

(ii) for almost every h E H the bounded measure on MJ^ 

(16) n^iE) = [ f{x)ri{h-\x) dx, E G B{W), 

has a density Uh G L^(R") for which 
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Under the above circumstances 



\{f,U,v)\'dg= / ( / My)\'dy 
G Jh 



dh 



a{h)l3{h)' 



Proof. Observe that Vth is the image measure, induced by $, of the bounded measure 
with density frj^ G L^(M"') with respect to dx (see e.g. Sec. 39 in [21]). Since VLh is 
bounded, the basic integration formula for image measures, (see Theorem C, p. 161 in 
j) and (pro]) imply that 



= f3-^h) [ e^^'^y^^Unhiy). 



Assume that J^\{f,Ugri)\'^ dg < oo. Since dg = Fubini's theorem implies that, 

for almost every h & H, 

\{f,U^a,h)V)\'da = m~' f \f e'-'^y'''Un,,{y)\^ da <+oo. 

This says that the inverse Fourier transform of flh is in L^(M"'), and the aforementioned 
Theorem (31.33) in [22] ensures that the latter condition is equivalent to saying that 
Qh has an L^(M")-density Uh with respect to dy. Furthermore, by Plancherel's theorem 

e^-''^y'''Un^,{y)\'da= [ \uHiy)\'dy. 



Applying again Fubini's theorem, fllSp follows and hence (IT7|) holds. Therefore (i) 
implies (ii). The converse statement is shown by applying the same argument back- 
wards. □ 

We are now in a position to state our first result. 

Theorem 1. IfU is a reproducing representation, then the image under $ of any Borel 
subset of with positive measure has positive measure. Hence 

(i) n < d; 

(ii) the set C of critical point^ of $ has measure zero. 

Proof. By contradiction, suppose that there exists a Borel subset A of with positive 
measure such that $(^4) is negligible. Since \A\d > and the Lebesgue measure is 
regular, there exists a compact subset K G A with li^'l^ > 0. Clearly, ^{K) is also 
compact, but |$(i^')|„ = 0. Take an admissible vector t] for U. The reproducing 
formula for / = xk and ffTSl) imply that 



0<\KU= [ ( [ My)\'dy] 
Jh \Jr" / 



dh 



a{h)(3{hy 



"^A point a; e M'' is critical for $ : M'' — >■ M" if the rank of the differential map is less than n. 
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SO that, on a subset of H of positive Haar measure we have ooh 0. Take then h G H 
such that Qh = ^hdy 7^ 0. Now, if E' is a Borel subset of M", the definition of Vth gives 

UhiE) = Qh{E n $(J^)) = [ uu{y)dy = 

JEr\'^{K) 

because = 0. Hence = 0, a contradiction. 

To show (i), assume that n > d and apply the above result io A = W^. Since $ is of 
class we have = 0, so that U cannot be reproducing. 

To show (ii), denote by C the set of critical points of $. Sard's theorem [52] implies 
that has measure zero. But then, by (i), also C has measure zero. □ 

3.2. Measures concentrated on the preimages under Given any x G M*^, let 
J($)(a;) = Y/det($^^7^"*$^ be the Jacobian of $ at x and denote by IZ the set of 
regular points of $, namely 

TZ = {xeW^ : J($)(x) > 0} = R'^ \ C. 

Lemma 2. The set TZ satisfies the following properties: 

(i) it is open; 

(ii) it is H -invariant and has H-invariant image under $; 

(iii) the restriction of $ to it is an open mapping; 

(iv) for every y in its image, ^~^{y) is a Riemannian suhmanifold ofW^; 

(v) a subset E C $(7?.) is negligible if and only if^~^{E) is negligible. 

Proof, (i) Since $ has continuous derivatives, TZ is an open set. (ii) The if-invariance 
follows from 

(19) ^^h..x{K-v) = h[(!>^^v], x,veM.'^, 

where h^, denotes the differential of the action x 1— )■ h.x and is therefore linear. Indeed, 
(1191) and the fact that u 1— )■ h[u] is a linear isomorphism, show that v G ker$^,j. if and 
only h^.v G kei^^h ^, so that dimker^^^. = dimkei ^^.h.x- Since x & TZ ii and only if 
dimker<l>*2^^ = d — n, the claim follows. To prove (fT9il . fix x G M'^, a tangent vector 
V G T^.(]R'^) ^ M.'^ and a smooth curve v{t) passing through x at time zero with tangent 
vector V. Evidently, h.v{t) is smooth and has tangent h^,.v at time zero. By (Q and 
again by the linearity of m 1— )■ h[u] 



t=0 



'dt 
as desired. 

Finally, (iii) and (iv) are standard consequences of the fact that, by definition of 
J($), the differential is surjective whenever x &TZ. 

In order to prove (v), put X = TZ and Y = ^{TZ). Since $ is a submersion from X 
onto Y and since X is locally compact second countable space, there exists a countable 
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family of diffeomorphisms : Ui x Vi ^ Wi such that {Wi} is an open covering of X, 
{Vi} is an open covering of Y, {Ui} is a family of open sets of M"^"" and 

(20) <l>(*,(^,y))=y {z,y)eUxV. 

Assume that E is a Borel subset of Y, then |$~^(_E')|(i = if and only if |$~^(_E') fl 
Wild = for all i. Since is a diffeomorphism, by the chain rule this is equivalent to 
|*7^($-i(E)nWi)|rf = 0, that is, hj (^,UiX (EnVi) is a negligible set of M^-"xM". 
Since Ui is an open non-void set, the last condition is equivalent to \E r\Vi\n = 0, that 
is, to \E\n = 0. □ 

Assumption 1 . Motivated by Theorem [H in the following we assume that C has 
Lebesgue measure zero. In particular, we assume that n < d. Furthermore, we fix 
an open iJ-invariant subset X of 7^ whose complement also has measure zero and we 
denote by Y its image under $, namely Y = $(X). Clearly, X satisfies the properties 
(i)-(v) described in Lemma [2] and so its complement is negligible. 

The next results are based on several kinds of disintegration formulae and their 
covariance properties with respect to the if-action. In Section lA.ll we review the 
general theory of disintegration of measures and introduce the pertinent notation. As 
for the induced if-action on measures, and the resulting covariance properties, we recall 
that, if 1/ is a measure on X and h & H, is the measure given by v^{E) = v{h.E) 
whenever E G B{X). Equivalently, 

(21) I ^{x)dv^{x)= I ip{h-\x) du{x) 

for every (f G Cc{X). The first disintegration we discuss arises from the Coarea Formula 
for submersions. 

Theorem 2. There exists a unique family {vy} of measures on X, labeled by the points 
of Y , with the following properties: 

(i) Uy is concentrated on ^^^{y) for all y G Y ; 

(ii) dx = Jy yydy; 

(iii) for any (f G Cc{X) the map y t— )■ dvy{x) E C is continuous. 
Furthermore, 

(22) = a{h)(3ih) Vy 
for all h E H and all y eY . 

Proof. The proof is based on the classical Coarea Formula. In Section IA.3I we give a 
short proof adapted to the situation at hand and we introduce the notation used in 
this proof. The reader is thus referred to Theorem [12] below. 

For every y E Y, define Uy by (175|) . Property (i) is then obvious and (ii) is the 
content of Theorem [121 

To prove (iii), fix G Cc{X) and yo G Y. If y^ ^ ^(suppv?), there is an open 
neighborhood V of yo such that V fl $(supp(/)) = 0. Thus ip{x) dvy{x) = for all 
y E V because Uy is concentrated on $~^(?/). If G ^(supp (p), taking a finite covering 
if necessary, we can always assume that there exists a diffeomorphism : U xV ^-^ W 
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such that (177|l holds, where U is an open subset of M*^"", V is an open neighborhood 
of yo and W is an open subset of X containing supp ip. The definition of Uy gives 



X 



ifix) duy{x) = / ip{<I/{z,y)){J<I/){z,y)dz 



and the map y i— )■ Jjj ip{'^{z,y)){J'^){z,y) dz is continuous on V by the dominated 
convergence theorem. 

In order to show (122]) . 'axh E H. Since the action of if on X is continuous, {j-'h[y]\y& 
is a family of measures on X and each of them is concentrated on as shown by 

4yy^{X \ ^'\y)) = urAviX \ ^-\h[y])) = 0, 

where the last equality is due to (i). Furthermore the family {i^h[y]}yGY is scalarly 
integrable with respect to dy because for all ip G Cc{X) 

(/ ip{x)diyjl[y]{x))dy = / (/ ip{h-\x) diyh[y]{x))dy 
Y Jx Jy Jx 

{ y ^ h-^[y] ) = a{h) / (/ ip{h~Kx) duy{x))dy 

Jy Jx 



= a{h) / Lp{h ^.x) dx 
Jx 

( X I— J- h.x ) = a{h)l3{h) I (p{x) dx, 

Jx 

where the third line follows from (ii). Hence 

dx= [ a{h-')P{h-')ujl^y^dy 
Jx 

and (iv) of Theorem [TT] implies that for almost all y E Y (122 p holds true. Item 
(iii) tells us that for any fixed ip G Cc{X), the mappings y H- ip{x) dPyi^x) and 
y I— 7- ip{x) du^^y^{x) are continuous and hence the almost everjrwhere equality is 
really an equality. □ 

In view of the previous result, we may apply the theory developed in Section IA.2I 
In particular we obtain (I73p in the case in which u and p are the Lebesgue measures: 

(23) L\X) = L\X, iyy)dy, f = fydy. 

Here the equalities must be interpreted in M{X) and the second integral is a scalar 
integral relative to the duality of M{X) and Cc{X). For a discussion of the details see 
the Appendix, where it is also explained that in particular 

|2 



(24) ||/r= / Wfytdy. 



Y 



One of the reasons for introducing the measures {uy} is because, via the coarea 
formula, they provide a very useful description of the density Uh discussed in Lemma [TJ 
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Corollary 1. Given f.rjE the function y H- {fy,Tiy)uy coincides almost every- 

where with the density ujh of the measure Qh defined by (fT6!) . 

Proof. Item (iii) of Theorem [TTl together with Theorem [21 applied to ff] G L^{X) and 
any ^ G Cc(Y) gives 

/ ^{^{x)) f{x)f]{h-\x)dx = / ^{y) / f{x)f]{h-\x)duy{x)dy. 
Jx Jy Jx 

The left hand side is nothing else but the integral ^{y) dVLh{y) because VLh is the 

image measure, induced by $, of frj^dx. The corollary follows. □ 

3.3. Reduction to fibers. Much of our analysis stems from decomposing the rep- 
resentation space L?{W^) in terms of the measures {vy}, and from a rather detailed 
understanding of the if-action on Y . We thus introduce the usual notation for group 
actions: if ?/ G F, then Hy is the stabilizer of y, H[y] = {h[y] : h G H} is the corre- 
sponding orbit and Y/H the orbit space. At this stage we need a hypothesis ensuring 
that the Y/ H is not a pathological measurable space. It is worth mentioning that this 
hypothesis is satisfied in all the significant examples that we are aware of. Below we 
further comment on this. 

Assumption 2. We assume that for every y & Y the if-orbit H[y] is locally closed in 
Y, i.e., that it is open in its closure or, equivalently, that H[y] is the intersection of an 
open and a closed set. 

The above assumption is not enough to guarantee that the orbit space Y/H is a. Haus- 
dorff space, hence locally compact, with respect to the quotient topology. However, it 
is possible to bypass this topological obstruction by choosing a different parametriza- 
tion of the iJ-orbits of Y. Indeed, a result of Effros (Theorem 2.9 in [T3]) shows that 
Assumption 2 is equivalent to the fact that the orbit space Y/H is a standard Borel 
space. Hence there is a locally compact second countable space Z and a Borel measur- 
able (hence Lebesgue measurable) map it : Y ^ Z such that 7r(?/) = TT{y') if and only if 
y and y' belong to the same orbit. To see this, observe that, by definition of standard 
Borel space, Y/H with the quotient cx-algebra is Borel isomorphic to a Borel subset 
of a Polish space Z. By Kuratowski's theorem [21], we may assume that Z = [0, 1]. 
Define 7r(y) = i{y), where y is the equivalence class of y in Y/H and i is the Borel 
isomorphism of Y/H into [0, 1]. 

In the following we fix the space Z whose points will label the orbits of Y and we 
choose on Z a pseudo-image measur^ A of the Lebesgue measure under the map vr. 
We note that A is concentrated on 7i{Y) and a subset E is A-negligible if and only if 
\7T^^{E)\n = 0, which is equivalent to |(7r o ^)~^[E)\d = (item (v) in Lemma [2]). 

Theorem 3. The following facts are equivalent: 
(i) the vector ri G L2(M'^) is admissible for U; 

^ It is a measure on Z whose sets of measure zero are exactly the sets whose preimage with respect 
to TT have measure zero in Y. It always exists since Y is CT-compact: it is enough to take first a finite 
measure on Y equivalent to the Lebesgue measure (just choose a positive density), and then to 
consider the image measure on Z induced by tt (see e.g. Chap. VI, Sect. 3.2 in [3]). 
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(ii) for X-almost every z & Z , there exists a point y G 7r~^(2) such that 

(25) \\<-t^^''^>Uf^§^y u^L^iX,.,). 
If (125|) holds true for y, then it holds true for every point in H[y]. 

Proof. Given 77 G L'^{X, dx), write 77 = fyVy dy where rjy G L'^{X, Uy). Fix y eY and 
put 

Vy = {ue L^X^Uy) : j^\{u.vi).f^^£^^ < +00}. 

The map Wy-.Vy^ L'^{H, a{h-^)(3{h~^)dh), defined by iWyu){h) = {u, r^J) for almost 
all G -ff, is a closed linear operator (the proof is standard [I3]). Hence it is enough to 
prove fl25l) for a dense countable subset of L'^{X, Uy). Hence we fix a countable family 
of functions {^pe} in Cc{X) with the following property: given an arbitrary ip G Cc{X), 
there exists a subsequence {ipejkm such that 

(26) \ipij < \ipo\, lim sup |v?4(x) - ip{x)\ = 0. 

The existence of such a family is clarified in Footnote \5\ in the Appendix. Clearly, for 
any y the family {(fe} is dense in L'^{X, Vy). 

Assume that U is reproducing and take an admissible rj G L^(X). For any i we thus 
have 

/ \{ViiUgVi)\^ dg = / \(pi{x)\^dx= / (/ \ip(,{x)\^ dvy{x)) dy, 

JG Jx JY JX 

the latter being a consequence of the coarea formula fl2^ . By Lemma [T] the measure 
^1 in flTBl) has an L^-density for almost every h E H and formula (ITS]) holds true; 
furthermore, Corollary [T] tells us that Uh can be expressed in terms of the measures 
{z/y}. Therefore 

\(fi{x)\^ duy{x) dy = / |(v9^, t/g?7)|^rf5( 
'y JX JG 

if \u^iiy)\'dy) 




H JY 



a{h)/3{h) 

where in the last line we have applied Fubini's theorem. Let A'^ C F be the set of 
y E Y where the equality 

(27) 

does not hold. Reasoning as in the proof of Corollary [H the equality of the first and 
last term of the above string is equivalent to saying that Ni is negligible. 
Put A^ = UfNi, a negligible set. For any y ^ N, ([57]) shows that {ipi} C Vy and 
Wy is an isometry on this dense subset. Since Wy is a closed operator, it follows that 
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Vy = L?{X, Uy) and (125|) holds true for every u G L'^{X, Uy). 

Now, is the set consisting of those y for which the equahty (125|1 does not hold 
for at least a m G L'^{X, Vy). We show that is //-invariant. Take h & H and y ^ N . 
For any G Cc(X), both </) and (p'* ^ are in L'^{X, Uy). Hence (125|) does hold for m = (y9 



and u 



Using (|2T]) and ([22]), we obtain 



|(^(x)|^rfZ//,[y](x) 



X 



\^{h.X)\' dV^^^y^{x) 

\Lp{h.x)\'^ a{h)l3{h)duy{x) 



X 



= a{h)P{h) 
{h.x = z) =a{h)l3{h) 



H 



X 



H 



X 



{hk = s) =a\h)f3\h) 



H 



ip{h.x)ri{k -^.x) duy{x 
^{zM{hk)-\z)du^-\z 

^{zMs-\z)du::-\z) 



dk 



a{k)/3{k) 
2 dk 



X 



a{k)(3{k) 
2 ds 



H 



^{z)f]{s~\z)a{h)P{h) dv^'\z) 



X 



ds 



a{s)l3{s) 



H 



X 



ip{z)ri{s ^.z)dv^y^{z] 
,o ds 



ds 



H 



that is, h\y] ^ A^, as desired. Finally, since is iJ-invariant and negligible, tt{UiN() is 
A-negligible and (ii) follows. 

The fact that (ii) implies that U is reproducing is proved by reversing the argument. 

□ 

Remark 4. Since vr induces a Borel isomorphism between the orbit space Y/H and 
7r(y), in the above statement and in the theorems of the following section it would be 
possible to avoid the space Z by considering ojiY/H a, a-finite measure defined on the 
quotient a-algebra, which, by Assumption 2 (Theorem 2.9 in [H]), coincides with the 
Borel (T-algebra induced by the quotient topology. However, this measure could fail to 
be finite on compact subsets. 

3.4. Disintegration formulae. Our next result. Theorem El is based on some clas- 
sical formulae that allow both a geometric interpretation of the integral (jUJ and a 
computational reduction that in the known examples is indeed significant. This is in- 
spired by the irreducible case, where it is known that U is reproducing (i.e. square 
integrable) if and only if the if-orbit, unique by irreducibility, has full measure and the 
inducing representation of the stabilizer Hy is square integrable [1]. 

We allude to formulae that express an integral over y as a double integral, first along 
the single i7-orbits and then with respect to the measure A on the space Z. Although 
these kinds of formulae can be traced back to Bourbaki |3] and Mackey [2S], perhaps 
one of the most famous occurrences of such a disintegration procedure appears in the 
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celebrated paper of Kleppner and Lipsman [23]; for a recent review see [19]. Much in 
the same spirit, we shall also need to decompose integrals over H by integrating along a 
closed subgroup Hq first, and then over the homogeneous space H/Hq, which we identify 
with a suitable orbit of Y. The topological hypothesis formulated in Assumption 2 is 
needed in order that these decomposition formulae can be safely applied. 

Recall that in the beginning of Section 13.31 we fixed a space Z that labels the orbits 
of Y and a measure X on Z whose null sets are in one-to-one correspondence with the 
if-invariant null sets of Y. 

Theorem 4. There exists a family {r^} of measures on Y, labeled by the points of Z , 
with the following properties: 

(i) Tz is concentrated on 7t~^{z) for all z G Z; 

(ii) dy = J^TzdX{z). 

Furthermore, for almost every z E Z the measure is relatively invariant and 
(28) t': = a{hy'Tz 

holds for every h E H. The family {r^} is unique in the sense that if {r^} is another 
family satisfying (i) and (ii), then = for almost every z E Z . 

Proof. The content of the theorem can be found in many different papers, such as 
Lemmas 11.1 and 11.5 in [2B] and Theorem 2.1 of [23], in slightly different contexts. 
The cited results are both based on Bourbaki's treatment of disintegration of measures. 
Here we simply adapt this theory to our setting. 

Theorem 2 Ch.VI § 3.3 of [3] yields a family {r^} of measures on Y labeled by the 
points z E Z, unique in the sense of the statement, such that 

• 7^ if and only if z E it{Y) 

• Tz is concentrated on ti~^{z) 

• dy = J^TzdX{z). 

The proof of Lemma 11.5 in [28] shows, under the circumstances that we are consid- 
ering, that for almost all z E Z (1281) holds true for all h E H; the density appearing in 
Lemma 11.4 of [22] is precisely a~^. □ 

3.4.1. A topological detour. Assumption 2 is needed in order to prove Theorem H] be- 
cause we apply results on disintegration of measures that use it, as developed in [3]. 
The same theorem actually holds under the (weaker) conditions that are described in 
the proposition below. Their equivalence does not seem to be a known fact. In |19j . 
Theorem 12, it is shown that (ii) in Lemma [2] below is a necessary condition for the 
disintegration in Theorem|4]to hold true. In the next statement tt denotes the canonical 
projection from Y onto Y/H. 

Proposition 2. The following two conditions are equivalent: 

(i) there exists an increasing sequence of compact subset {Kn} of Y such that the 
complement ofUKn is Lebesgue negligible and if{Kn) endowed with the relative 
topology is a Hausdorff space; 
(ii) there exists an H -invariant null set N gY such that {Y \ N)/H is a standard 
Borel space with respect to the a-algebra induced by vr. 
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Proof. First we show that (i) imphes (ii). Denote by R the equivalence relation induced 
by the action of if on y, that is, y y' if and only if 7i:{y) = Ti{y')- 
Claim 1: there exists a Lebesgue measurable map p from Y into a locally compact 
second countable space VL with the property 

(29) p{y) = p{y') y ~h y'. 

By assumption for each n the space Tf{Kn) is Hausdorff and, by Prop. 3 Ch.l § 5.3 of 
[2], this is equivalent to the fact the quotient space K^/ Rn is Hausdorff with respect to 
the quotient topology, where Rn the restriction of R to Kn x Kn- Since Y is cx-compact, 
the above property implies that is a Lebesgue measurable equivalence relation ac- 
cording to the definition in Ch. VI § 3.4 of [3]. By Proposition 2 Ch. VI § 3.4 of [3] 
there exists a map p -.Y ^ VL with the desired properties. 

Claim 2: for any compact set K of F, the set H[K] is Borel measurable. Indeed, 
since H is a-compact, there exists a countable family {Hm} of compact subsets of H 
such that H = UmHm and, hence, H[K] = UmHm[K]. Hence H[K] is countable union 
of compact subsets, hence Borel measurable, since the action of if on y is continuous 
and Hm x K is compact. 

Claim 3: there exists an if-invariant Borel set Yi whose complement is Lebesgue neg- 
ligible and such that the restriction p^y^ is Borel measurable. The proof of Proposition 2 
Ch. VI § 3.4 of [3] actually irnplies the claim. For completeness, however, we present a 
direct proof. Lusin's theoremlj yields an increasing sequence of compact subsets {K'^} 
of Y such that the complement of UK'^ is Lebesgue negligible and the restriction of 
p to each Km is continuous. By Claim 2 the set Yi = H[UmKm] and its complement 
A^^i = y \ Yi are both if-invariant Borel subsets, and A^^i is Lebesgue negligible since 
Ni C Y \ UmKm- To prove that p\Yi is Borel measurable, for any closed subset C C Q 

p^^^iC) = p-\C) n Fi = Ump-\C) n H[K'm] 
= yjmH\cnK'j = vjmH[p-^jC)l 

since p~^{C) = H[p~^{C)] by fl29|) . Since P|^^(C) is compact. Claim 2 implies that 
P|Y^(C) is Borel measurable. 

Claim 4: the quotient space Yi/H is analytic. Since Yi is a Borel subset of a locally 
compact second countable space, it is standard and, hence, analytic. By Theorem 5.1 
of [29] , if a quotient space of an analytic Borel space is countably separated, then it is 
analytic. Hence, it is enough to exhibit a countable family {A^} of if-invariant Borel 
sets of Yi with the property that for any pair of points y, y' G Yi such that y t^/j y', there 
exists Am such that y G Am and y' ^ Am- To find such a family, choose a countable 
base {Vm} for the second countable topology of Q and define Am = P^-^iVm), which is 
an if-invariant Borel subset of Yi by (l29l) and Claim 3. If y 7^/? y', then p{y) 7^ p{y') 
and, since fl is Hausdorff, there exists Vm such that p{y) G Vm and p{y') ^ Vm, that is, 
y e Am and y' ^ Am- 

Claim 5: there exists an if-invariant Borel set Y2 C Yi whose complement is Lebesgue 
negligible and Y2/H is a standard Borel space. Since Y is second countable, there exists 
a finite measure on the analytic space Yi/H, which is the pseudo-image measure of the 



See, for example, Theorem 5.6.23 j31j or the definition of measurable function given in [3]. 
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piy) 



Lebesgue measure of Y. By Theorem 6.1 of [29], there exists a Borel subset E C Yi/H 
whose complement is neghgible and E is a. standard Borel space. The set Y2 = 7r~^(£') 
has the desired properties. 

Item (ii) is proved by setting N = Y \Y2 = Ni U {Yi \ Y2) and observing that 
(y \ N)/H is Borel isomorphic to E. 

We now show that (ii) implies (i). By assumption there exists a Borel //-invariant 
Borel set N G Y with zero Lebesgue measure such that {Y \ N) /H is Borel isomorphic 
to a Borel subset of [0, 1] and, hence, there exists a Borel injective map j : (Y\N)/H — t- 
M. If 7^ 0, fix a section s : N/H N,a point yo e Y\N, and define p lY ^ Yx[0,l] 
by 

\yo,i{7r{y))) y^N 
{s{n{y)),0) yeN. 

Clearly, the map p is Lebesgue measurable and p{y') = piy) if and only if ixijj) = n^y'). 
Lusin's theorem implies that there exists an increasing sequence of compact subsets 
{Km} such that the complement of UKm is Lebesgue negligible and the restriction of 
p to each Km is continuous. By a standard result in topology, (see e.g. Corollary 1 
of Proposition 8 § 10.6 of [2]), Ti{Km) is homeomorphic to p{Km) which is a compact 
subset of a Hausdorff space, so it is Hausdorff. □ 

In the statement of the above proposition Y can be replaced by any locally compact 
second countable space, the Lebesgue measure by a measure on Y and the equivalence 
relation induced by H by any other equivalence relation. 

3.5. The integral decomposition of U. From now on Assumptions 1 and 2 are 
taken for granted. The main result here is that Theorems [2] and HI which hold both 
true, yield an integral decomposition of the mock metaplectic representation in terms 
of induced representations of the isotropy subgroups of H. This fact, which is of 
independent interest, is at the root of Theorem El where the admissible vectors for U 
are characterized. 

Proposition 3. For almost every z & Z the family of measures {uy} is scalarly inte- 
grable with respect to Tz, the measure on X 



Y 



is concentrated on the H-invariant subset $ ^(tt ^{z)) and for all h ^ H 

(30) = mf^z- 

Furthermore, the family of measures {fJ^z} is scalarly integrable with respect to A and 

(31) dx = / fj,2 dX{z). 



z 



Proof. The map tt o $ is a Lebesgue measurable map from X to Z and A is a pseudo- 
image measure of the Lebesgue measure restricted to X under tt o $ by construction 
of A and Assumption 1. Hence, Theorem 2 Ch. VI § 3.3 of [5] yields a family {fiz} of 
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positive measures on X such that each /i^ is concentrated on $ ^(vr ^{z)) and, for all 
(32) / (p{x)dx = / ( / ^{x)d^z{x))d\{z). 

J X J Z J X 

For any fixed if G Cc(X), y h-> j^{p[x)duy{x) is Lebesgue integrable by (ii) of The- 
orem 121 Hence, appealing to (ii) of Theorem S] and to (iii) of Theorem [TTl we know 
that for almost all 2; G Z, the map y 1— )■ ip{x)duy{x) is r^-integrable, the map 
z I—)- ^y{^-^ip{x)di'y{x))dTz{ii) is A-integrable, and 

( / ( / ip{x)di'y{x))dTz{y))d\{z) = / (/ (p{x)diyy{x))dy = / ip{x)dx. 

IZ JY Jx JY Jx Jx 

Comparing this with fl32p we infer that for almost every z ^ Z 



(33) / (/ ip{x)diyy{x))dTz{y) = / V5(x)rf/i^(x). 

JY JX JX 

The set of 2; G Z where the above inequality does not hold is A-negligible and, 
can be chosen independently of (f. Indeed, as explained in Footnote \5\ we may find a 
countable subset S of Cc{X) such that, for any ip G Cc{X), there is a sequence (ipi) in S 
converging to if uniformly and \ipi\ < \ipo\ for all i. For each cp & S there is a negligible 
set C Z such that the map y ^ fx V^(^) dvy^x) is integrable with respect to Tz for 
all z ^ N^p. Denote by the A-negligible set V^^p<zsN^p. We now claim that the family 
{uy} is scalarly integrable with respect to for all z ^ N . Indeed, given ip G Cc(X), 
there is a sequence [tpi) in S converging to ip uniformly and \ipi\ < |v?o| for all 2. Write 
([3SD for each (yjj. Since \ipi\ < Iv^ol we may apply the dominated convergence theorem 
to the right hand side. As for the left hand side, for the same reason we may apply the 
dominated convergence theorem to the inner integral. Further, since y h- )■ ^'^^(supp ipo) is 
r^-integrable we may apply dominated convergence to the outer integral. The claimed 
independence of ip is proved. 

Hence for all z ^ N , the family {vy} is scalarly integrable with respect to and 
IJiz = Jy dTziy). Finally, 'ax z ^ N and h E H. For all ip G Cc{X) 

ip{K'^ .x)d^z{x) = / I / ip{h~'^ .x) dvy{x)] dr^iy) 



X JY \JX 



a{h)(3{h) I yj ip{x) duh-i[y]{x) ] dTz{y) 



= (3{h) J yj ip{x)duy{x)j dTz{y) 

= (3{h) / ip{x)dfiz{x) 
Jx 

where the second line is due to the change of variables x t— h.x and fl22]) . and the third 
line to y I— )■ h.y and (1251) . This proves that /i^ = f3{h)fiz. □ 

By virtue of Proposition 131 we may consider the Hilbert space L'^{X,fiz) for almost 
every z E Z. Whenever /x^ is not defined, we redefine = and /i^ = 0, and set 
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/i^) = {0}. Proposition [6] below, or equation (173|) . both based on Proposition |3l 
will allow the following Hilbert space identifications 



(34) L\X)= / L^(X,/i,)dA(z) /= / hdXiz) 

J z Jz 

(35) L\X,fi,) = J^L\X,Uy)dTM fz = j^UydrM, 

where / G L'^{X), G L'^{X,fiz) for all z e Z and, fixed z, f^^y G L'^{X,i'y) for 
all y E Y. The integrals of Hilbert spaces are direct integrals with respect to the 
measurable field associated with Cc{X), and the integral of functions are scalar integrals 
of vector valued functions taking value in M{X). Indeed, as explained in the Appendix, 
we shall regard L^(X), L'^{X,fiz) and L^(X, z/y) as subspaces of M[X) in the natural 
way. In particular, if / G Cc{X), fz is the restriction of / to $~^(7r~^(2;)) and fz^y is 
the restriction to $~^(y). Furthermore, for any / G L'^{X) 



(36) ||/r= / / \\Uy\\t^dTziy)dXiz). 

Jz Jy 



Formula fl34l) induces the following decomposition of U. 

Lemma 3. The representation U is the direct integral of the family {Uz} of represen- 
tations acting on L'^{X,fiz) by 

{UzJ){x) = /3(/.)-5e-2'^'<*(^)''^>/(/^-'.a:) 
for g = {a, h) E G and f G L^{X, Hz) ■ 

Proof. For each 2; G Z, the map g 1— ?■ Uz^g is a strongly continuous unitary representation 
of G by the same proof of Proposition [1] since /x^ and the Lebesgue measure are both 
relatively invariant with the same character /3, (compare ([7]) with (l30i) ). We now 
prove that {Uz} is a A- measurable field of representations. Indeed, for any g E G and 
^,^'eC,{X), 

{Uz^,^'),^ = [ /3(/i)-^e-2-<*(-)''^)(^(/.-i.x)7Rc^/i.(x). 



X 



Since x ^ e'^^'^'^'^''^''''^ ^{h-\x)ip'{ x) is a compactly supported continuous function and 
the family {/i^} is A-scalarly integrable, the map x 1— )■ {Uzf, ^')^^ is A-integrable, hence 
A- measurable. 

Finally, to prove that U = J^Uz dz it is enough to test the equality on Cc{X). For any 
g E G and ip G Gc{X), we regard Ugip and Uz,g<p as elements of M(X). Hence, fl3T]) 
gives 



Ug(p -dx = {Ug(p ■ ^iz)dz = / {Uz,g(p ■ iJ^z)dz 
Jz Jz 

by definition of Uz- □ 



The next technical lemma is needed in order to prove that Uz is equivalent to an 
induced representation. 
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Lemma 4. Fix y eY and h E H. The map Ty^h '■ L'^{,X, Uy) — )• L'^{X, Uhiy]) defined for 
Uhiyj-almost every x E X by 

(r,,,/)(x) = v/«(/i-i)/3(/i-i)/(/i-^x) 
is a unitary operator. Furthermore, for every h,h' E H and every y EY 

(37) Th[y]^h'Ty^h = Ty^h'h 

(38) Tyl = Th[y]^h-i. 

Proof. Given a Borel measurable function / which is square-integrable with respect to 
Uy, the map x i— )■ {Ty hf){x) is also Borel measurable and it is square-integrable with 
respect to Uhiy] since 

aih-')(3ih-') [ \fih~\x)\'du,[y]ix)= [ \fix)\'duyix), 
Jx Jx 

by the change of variables x h- )■ h.x and fl22|) . The above equation implies that Ty^h is 
a well-defined isometry from L'^{X, Py) to L^(X, Uhiy])- Equality (137|) is clear and, as a 
consequence, Th[y]^h-^Ty^h = ^y,e is the identity on L^{X, i^hiy]) so that Ty^h is surjective, 
thereby showing ( 138|) . □ 

For any 2; G vr(y), we fix an origin yo in fhs orbit 7r~-'^(z) = iJ[?/o] and we denote by 
Hz the stabilizer at yo- We denote by /C^ = L'^{X, Vy^). 

By fl22p we know that Vy^^ is relatively invariant under Hz- It follows that it make 
sense to look at the quasi-regular representation of acting on /C^, whose value 
sX s E Hz is Kz,s = Tyg^g. As usual, we extend A^ to a representation of M" xi Hz by 
setting Az^a = e~^'^*^^°'"^ id for all a E M". Finally, we denote by Wz the representation 
of G unitarily induced by A^ from M" xi Hz to G. We realize Wz as a representation 
acting on the space Tiz of those functions F : G ^ Kz that satisfy 

(Kl) F is (igf-measurable; 

(K2) For all ^ e G and (a, s) eW ^ Hz 

F{gas) = ^^{P^Kl,F{g)- 

(K3) := j^\\F{h{y))\\l^a{h{y))dTz{y) < +00. 

Here h{y) E H is any element in H that satisfies /i(2/)[yo] = y for ^2- almost all y eY. 
Since is concentrated on H[yQ], it is enough to define h{y) for y E H[yo] and, due 
to the covariance property in (K2), the integral does not depend on the choice of 
h{y) in the coset kHz- Furthermore, (K2) implies that it is enough to know these 
functions on H. Two functions F and F' are identified if — = 0. The induced 
representation on Hz is defined for g E G by the equality 

{Wz,yF)ig') = Fig-'g') 

valid for dg-almost every g' E G. 

For the sake of precision, if z E Z \ n{Y) we put Uz = 0, /C^ = {0} and Hz = {e}; 
recall that = and that X{Z \ 7t{Y)) = 0. 
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Lemma 5. Fix z E Z such that Tz 7^ 0. The map Sz ■ L'^{X,fiz) — ^ 'Hz whose value at 
fz = JYfz,ydTz{y) is given by 



is a unitary operator intertwining Uz with Sz- 

Proof. For any {a,h) E G, fz,h[yo] ^ L"^ ^ ^h[yo]) ■ Hence T~^hifz,h[yo]) ^ ^z- In order 
to prove that Szfz is c/gf-measurable it is enough to show that 



is (i/i-measurable for every ip G Cc{X) because Cc{X) is a dense subspace of the separa- 
ble Hilbert space /C^. Since fz = Jy fz,y dTz{y), there exists a square-integrable function 
/ : X — >■ C and a r^-neghgible set C y such that, for all y ^ N, f belongs to the 
equivalence class of fz,y G L^{X,h'y). Define N' = {h E H \ h[yQ] E N}, a negligible 
set with respect to the Haar measure dh because, by fl28l) . Tz is non-zero relatively 
invariant on the orbit H[yo]. Then for all h ^ N' 



which is clearly d/i-measurable. Next we prove the covariance property (K2). For 
g = (a, h) = ah and (fe, s) = 6s e R" x Hz, 



{Szfz){a,h) = .MlF^ e 



'^yo,hifzMyo]) 



h ^ {Tyo]hifzMyo])^f)ic. = \/a{h)(3{h) / fz^h[yo]{h.x)ip{x)duy^{x) 



X 




{Szfz){ahbs) 



iSzfz)ia + h^[b],hs) 



^a{h~^)a{s~^) e2-<'^^[^ol''^+'^n^> T-%iUsM) 



g2..(M.o],/.tM) T^-^i^^Szfz){a, h) 



^/^(^ e2..(,o,.> K~l{^Szfz){a,h) 



by definition of h'^ and A^. Further, 



jj\{Szfzmy))\\l^a{h{y))dTz{y) = jj^T;;^^^^^^^ 




whence (K3). This also shows that 5*^ is an isometry from L'^{X,fiz) into H 
Finally we prove that 5*^ is surjective. Given F E Tiz, for aA\ h E H define 



fz,h = Ty,,h{F{h)) E L\X, Uhiy,i). 

Since F satisfies (K2), it follows that fzAs = fz,h- For ip E Cc{X) the map 



h^,Mh){Ty,,,{F{h)),^),^.. = v^(F(/i),/)^, 
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is (i/i- measurable since h H- F{h) is (i/i- measurable from H into /C^ and the map 
h I— 7- a{h)ip^ is continuous from H into /C^- Therefore 



\\f.My)\\lydrM = JJ\F{h{ymla{h{y))dTM < +00. 

It follows that = Jy fzMy) d-'^ziy) is in jy L'^{X,Vy) dr^{y) = L'^{X,ij,^) and, by 
construction, S^fz = F. 

Finally, we check the intertwining property on the dense subset Cc{X) of L^(X). If 
g = a ^ M", for any if G Cc{X) and for almost every h ^ H 



= {S^(f){-a, h) = {S^ip){a~^h) 



where, in the second line, we have used <I>(x) = /^[yo] for z//i[j,g]-almost every x E X . If 
g = keH, 



= Va((A;-i/i)-i) (T^[j^„],fc-iTy„,/,)"V 

Since two functions in "H^ that are equal for almost every h E H, are equal almost 
everywhere in G, the intertwining is proved. □ 

Recall that L'^{X) = L'^{X, fiz) dz, where the direct integral is defined by the 
measurable structure associated with any fixed dense countable family {(fik} in Cc{X). 
Clearly, z H- {Szfk} is a measurable structure for the family {Hz}, and we define the 
direct integral 7/ = f^Hz dz. 

Theorem 5. The map S : L'^{X) 

Sf = Szfzdz f = / fzdz 



is a unitary map intertwining the mock metaplectic representation U with the unitary 
representation W of G acting on Ti given by 

W= f Wzdz. 



Proof. The statement follows from the definition of the measurable structure for the 
direct integral J^Tizdz, from Lemma [3] and Lemma [51 □ 
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3.6. Admissible vectors. We are in a position to state our main result. We need, 
however, a last disintegration formula, sometimes referred to as Weil's formula (see 
e.g. [13 ), a rather straightforward consequence of the theory of quasi-invariant mea- 
sures on homogeneous spaces. The easiest way of formulating it is perhaps that for 
any G Cc{H) the following integral formula holds 

(39) ^ ^{h)a{h-')dh = J^Q^ ^{hiy)s)ds^ drM. 

where ds is a suitable Haar measure on the stabilizer and where as before hiy) G H 
is any element that satisfies /i(2/)[yo] = U for r^-almost every y eY . We interpret fl39|) 
along the same lines of thought that we have followed for the other formulae by writing 

(40) a-^ ■ dh = ! {dsf^y^"' dT,{y) 



Y 



as an equality of measures on H. This time ds is regarded as a measure on H concen- 
trated on Hz, so that the translated measure {ds)^^^^ is concentrated on h{y)Hz. As 
usual, we shall extend fl39|) to L^-functions by means of Theorem [11] By Theorem 2 
(and the comments below) in Ch. VII § 3.5 of for all s E the modular functions 
of H and are related by the formula 



(41) a-\s 



Theorem E] establishes that U and W are equivalent. Therefore, we formulate our 
necessary and sufficient condition for the existence of admissible vectors of U for those 
of W. Thus, any admissible vector F E % for W is to be thought of as the image 
under 5" : L'^{X) — > "H of an analyzing wavelet rj. 

Theorem 6. The function F = j F^dX^z) is an admissible vector for W if and only 
if for almost every z E Z and for every m G /Cz = L'^{X, Vy^) 

(42) ||u||^^= I ([ \{u,AzjFzA~^'/'){h{y)))^fdsy{h{y))drziy). 

J J Hz 

Proof. By the definition of T given in Lemma |H for every h E H and yo eY 

Tyo,h-^{v^)z,yoix) = \/a{h)l3{h){r]^)z^h-^[yo]ih.x) = \/a{h)l3{h)riz,h-^yo]ix) 
holds for any rj = f^fy Vz,y dr^iy) d\{z) E L'^{X) and hence 



Suppose now that rj is an admissible vector for U or, equivalently, that F = Srj is such 
for W. By Theorem El what we have just established and the definition of S given in 
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Lemma [5], for almost every 2 G Z and any fixed yo G ir^^^z) 



(43) 



ML 



H 



H 



H 



H 



H 



(^,(^').,.o)l 



dh 



a{h)(3{h) 



{u, ^a{h)l3{h) {Ty^,h-^) ^ r/^,h-i[y(,]) p 



dh 



a{h)P{h) 



{u,F,{h))\^AH{h~^)a(h) dh. 



dh 
a{h) 
dh 



Hence, applying (139|) . the covariance property (K2), ( 14T]) and ([5]) we obtain 



\u\ 



Y \JH 



\{u,Fihiy)s))[' 



a^{h{y)s) 
^H{h{y)s)) 



ds dT^{y) 



Y \JH. 



Y \JH, 



|(m,A,,,-iF(%)))| 



^H{h{y)) 



\{u,K^,F{h{y)))\ \ 
Y \JH^^ ^G{h[y)) 



ds a{h{y))dT^{y), 



which is fH21) . Conversely, if fH2]) holds for some F El-i, then reading the above strings 
of equalities backwards yields the first line in Therefore, by Theorem [31 77 is 

admissible for f/, hence F is such for W . □ 

Corollary 2. Assume that U is a reproducing representation and suppose that z & Z 
is such that fl42l) holds true. Then: 



(i) if^^^^yo) is a finite set for some y^ G ^^^{z), then the stabilizer Hy is compact 

for every y G ti"^{z); 
(a) if G is unimodular and the stabilizer Hy is compact, then (^~^{y) is a finite set, 
hence n = d. 

Proof. Clearly, it is enough to prove (i) and (ii) for the origin y^. Take a (countable) 
Hilbert basis {ui} of /C^. Apply fH2|) to each element of the basis and sum 



dim/C, 



Y ^JH^ 



Y,\{n.,Ks {f.^g"^) {h{y)))^fds)a{h{y))dTM 



(44) 



Y ^JH, 

ds 



1/2 



{h{y))\\iJs)a{h{y))dT^{y) 



F,A-'/\h{y))\\la{hiy))dTM- 



Y 
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Now, if <l>~^(i/o) is a finite set, then the left hand side is finite and strictly positive, 
hence so is the right hand side, so Hz has finite volume. This proves (i). If Ac = 1 and 
Hz has finite volume, then the right hand side is finite and strictly positive by (K3). 
Hence $~^(?/o) is a finite set and since it is a regular submanifold of dimension d — n, 
necessarily n = d. Thus (ii) holds. □ 

3.7. Compact stabilizers. As a preliminary step, we assume that the stabilizer Hz 
of a given z & Z is compact, hence such is any other stabilizer in the same orbit. Later 
we shall assume that this is the case for almost every orbit. 

The compactness of the stabilizer allows us to use Schur's orthogonality relations for 
computing the inner integral over Hz in fH2]) . Indeed, since Hz is compact, the repre- 
sentation is completely reducible. Hence, for each equivalence class s in the dual 
group Hz, we can choose a closed subspace ]Cz,s C /C^ such that the restriction A^^^ 
of Az to JCz^s belongs to s, and we denote by the multiplicity of s in A^ (with the 
convention that JCz^g = if mg = 0). The following direct decomposition in primary 
inequivalent representations holds true 

(45) /c, ^ /C,,, ® C'"^ A,~0A,,,®id, 

where we interpret C™^" = i"^ whenever = Kq. Furthermore, for any cardinal m G 
{!,..., No}, we denote by {ej}^^^ the canonical basis of C™. 

Mackey's theorem on induced representations of semi-direct products [28] guarantees 
that each induced representation Ind^dx,j:^^(e~^'^*^^''''^ A^^s) is irreducible on Hz,s and 
gives the following direct decomposition in primary inequivalent representations for 
Wz-. 

(46) ~ -Hz,!^ ® C™^ Wz-^ Ind^.,^^(e-2-<^«'-> A,,,) ® id . 
By (145|) and (146|) . respectively, we have 

U = ® Cj, M G JCz- 

We write vol Hz for the mass of Hz relative to the unique Haar measure ds that makes 
formula f l39p work. Note that vol H^ is not necessarily one. 

Proposition 4. Let z & Z be such that the stabilizer Hz is compact. Given Fz G Hz 
the following facts are equivalent: 

(i) equality ( 142|) holds true for all m G /C^; 

(ii) for all s E Hz such that mg 7^ 0, and for all i,j = 1, . . . ,ms 

(47) l(FM), F^MHy))K..^^^irAy) ^ '-^s,. 
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Proof. Take u G /C^. We compute the inner integral in ( 142|) using Schur's orthogonality 
relations. For r^-almost every y 

nig 



' dim /Cj 

Choosing u = us^i, fj42l) is equivalent to 



(48) IWF.MHvml.^W^drM-^"''' 



Choose next j i and u = Ug^iQ) ugj- Taking ( 148|) into account, (142|) is equivalent to 

Hence (i) is equivalent to (ii). □ 

Equation fHTl) has the following interpretation in terms of the abstract theory de- 
veloped by Fiihr [18]. Indeed, for each irreducible representation of G in (146|) . we can 
define the (possibly unbounded) operator dz^s on "H^^^ 

(49) dz,,FzAg) = ^^J^Acig) F^M^ 

which satisfies (K2) precisely because the stabilizer is compact. The operator d^^s is a 
positive self-adjoint injective operator semi-invariant with weig ht [B]. Now, (HZD 
says that F^^s^i is in the domain of d'^"^ and 

(50) {d'lJ'^Fz.s.i, d~^/'^Fz,§j)'H,,s = ^iv = 1, • • • , T^s- 

One should compare this with Theorem 4.20 and equations (4.15) and (4.16) of |18j . 

Corollary 3. Let z E Z he such that the stabilizer Hz is compact. The following are 
equivalent: 

(i) there exists Fz G Hz such that equality (142|) holds true for all u E KLz', 

(ii) 'mg < dim('H^_j) for all s E Hz. 

If G is non-unimodular, this last condition is always satisfied. 

Proof. Fix s E Hz such that 7^ 0. If G is unimodular, dz^s is the identity up to a 
multiplicative constant, so that the families {Fz,s,i\^!^i satisfying (150|) are precisely the 
orthogonal families in l-iz^s with square norm equal to dim/C^,^/ vol//^, whose existence 
is equivalent to mg < dim('H2 ,5). If G is non-unimodular, dz^s is a semi-invariant 
operator with weight A^^^. Therefore its spectrum is unbounded (see formula (2) of 
so that dim'H^ ,? = +00, provided that m§ 7^ 0. Hence the families {Fz^s,i}^i 



satisfying fl50l) are the families in the domain of d^ - that are orthonormal with respect 

— 1/2 

to the inner product induced by d^ ~ . □ 
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If G is unimodular, (ii) of Corollary |2] implies that /C^ is finite-dimensional, so that 
mg = for all but finitely many s E for which is finite. Furthermore, the orbit 
T[~^[z) is often infinite, so that dimT/^.s = +00 and the requirement < (lim.'Hz,s is 
trivially satisfied for every s & H^- 

From now on we assume that almost every stabilizer is compact. For each z 
we can thus apply Proposition |4j Theorem [7] provides an explicit decomposition of 
the representation W , hence of ?7, as a direct integral of its irreducible components, 
each of which is realized as induced representation of the restriction of to a suitable 
(irreducible) subspace. The result does not depend on the fact that U is reproducing. 
To state the theorem, we fix a Borel (hence A) measurable section o : vr(y) — )■ Y whose 
existence is ensured by Assumption 2 and by Theorem 2.9 in [T3], thereby choosing 
o{z) as the origin of the orbit 7r~^(z). We then extend o : Z ^ Y measurably. Thus, 
for all z E Z,we have /C^ = L^(X, z^o{^))- 

Lemma 6. The field of Hilhert spaces z ^ ICz is X-measurable with respect to the 
measurable structure induced by Cc{X) C Kz and the corresponding direct integral 
K, = j^K,zd\{z) is a separable Hilbert space. 



Proof. For any ip,(p' G Cc{X) the map z h- )■ ip{x)ip{xydh'o(z){x) is A-measurable 
because y 1— )■ j-^ip{x)ip{x)'duy{x) is continuous (see (iii) of Theorem [2j) and is Borel 
measurable. Since Z is second countable. Corollary of Proposition 6 Ch. II § 1.5 in [TT] 
implies that K, is separable. □ 

Theorem 7. Assume that for \- almost every z E Z the stabilizer Hz is compact. There 
exist a countable family {z 1— >■ /C"}„g_\r of X-measurable fields of Hilbert subspaces IC^ 
of Kz, and a family of cardinals {mn\n&N C {!,..., Kq} such that, for almost every 
zeZ, 

(51) lCz = @lCl®C^\ 

neJ\f 

(52) = 55 id, 

where fl52|) is the decomposition of Az into irreducibles. 

Before the proof, some remarks are in order. 

Remark 5. In ( 15T|) it is understood that, for each n E N and j = 1, . . . , mn, the field 
of Hilbert subspaces z ^ /C" <S) C{ej} is A-measurable. 

Remark 6. For each n E M and for almost every z E Z we denote by T-Vl the Hilbert 
space carrying the induced representation IndiRdx,/^^(e~^'^*^°^^''''^ A"). Reasoning as in 
the proof of Theorem 10.1 of [2E], for each n E J\f, z 1— )■ "H" is a A-measurable field of 
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Hilbert subspaces, "H" C "H^, and 

(53) H = /"k rfA(z) ®C™" 

(54) ^ = / Indi,.,^^,(e-2-^<''(^)' > A^) dX{z) ® id 

where, by Theorem 14.1 of [28], each component IndKdxij/^(e~^'^*^°*^^''''^ A") is irreducible 
and two of them are inequivalent provided that they are different from zero (see the 
next remark). 

Remark 7. In the statement of Theorem [71 given n G A/", it is possible that for some 
z E Z the Hilbert space /C" reduces to zero as well as "H". If this is the case, then 
clearly A^ and lnd^d^H^{e-^'''^°'^'^'-^ Xfj can be removed from the corresponding integral 
decompositions of A^ and W . 

Remark 8. Fix z and compare (HSj) with flS2]) . The set A/" is a parametrization of the 
relevant elements in the dual group defined by the direct decomposition of A^ into 
its irreducible components A". In other words, for each n G A/" for which /C" ^ 
there exists Sn G such that A" = A^^^^ and = is its multiplicity, which is 
independent of z by its very construction. 

Remark 9. As a consequence of Theorem [7] and general results on direct integrals, for 
each n E Af there exists a A-measurable field {z £^i}e>i of Hilbert bases for each 
field z I—)- "H" and, for any F eT-L, 



(55) ^=EE / F:,dX{z)0e, 

neAf j=l 



where z i— )■ f^i{z) is a A-measurable complex function and 

(56) \\nl = Y.Ilf KMn,dX{z) = [ \m')\'dX{z). 

n&N j=i ^ neM j=i e>i 

Conversely, if {z i— f'j[i{z)}n,j,i is a family of A-measurable complex functions such that 



then (1551) defines an element F El-i. 



EEE / \fU^)\'dX{z) <+oo, 



of Theorem We claim that there exists a sequence of Borel measurable functions C,k '■ 
Y ^ H such that, for any y eY, the set {^k{y)}k£N is dense in Hy. To this end, define 
YxH^YxYhj S(y, h) = {h[y], y), a continuous map, hence Borel measurable. 
Now, the diagonal D = {{y,y) \ y E Y} is a Borel set and Hy = {h E H \ E{y, h) E D} 
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for any y ^ Y. By Aumann's measurable selection principle (see e.g. Theorem III. 23 
of [6]) the desired sequence exists. 

For all z G Z, let C CiJCz) denote the von Neumann algebra on /C^ generated by 
the representation Koi^z) of Ho(z)- We show that z i— ?■ is a A-measurable field of von 
Neumann algebras. For each z & Z the continuity of s H- implies that the family 
{Az^^,^(o{z))}k€n generates M^. Hence, it is enough to prove that for any k & N the field 
of operators z H- A^^^i^^oiz)) is A-measurable. This means that for any ip,ip' e Cc{X), 
the map 



z 



v/a(a(o(z))-i)/3(a(o(^))^^)^(a(o(^))-^a;)^'(x)rfi.„(,)(a;) 

'X 

is A-measurable. First we claim that 



Jx 



is continuous onY x H. Fix [yo, h^) &YxH and e > 0. By (iii) of Theorem |2] applied 
to (p^"(p' G Cc{X) there exists a compact neighbourhood U of yo such that for ally & U 



(p{hQ^ .x)(p'{x)diyy{x) — / (p{hQ^ .x)(p'{x)diyy^^{x)\ < e/2. 
X Jx 

Choose a compact neighbourhood V ol ho e define K = V. supp (f, which is a compact 

subset of X. The map y H- {i^y)K is continuous from U to M{K) = C{K)* with respect 

to the weak* topology, so that sup^^u Uy^K) is bounded (Corollary II. 4 of [5]). Now, 

the map h ip'^ is uniformly continuous. Hence there is a compact neighbourhood 

V <ZV oi ho such that, for all h G V, ip{h~'^.x) = ip{h^^.x) = if x ^ K and 

The triangular inequality gives that for all (y, /i) G t/ x V 



(p{h ^.x)(p'{x)diyy{x) — / (p{hQ^.x)(p'{x)diyy^{x)\ < 
X Jx 



so that the claim is proved. Since h a/ a{h)~^l3{h~^) is continuous and z i— )■ {o{z) , ^k{o{z))) 
is Borel measurable from Z toY xH,it follows that z i— )■ Az^s,k{o{z)) is a Borel measurable 
field of operators and, hence, A-measurable. 

Proposition 1, Ch II § 3.2 of [H] shows that M := dX{z) is a von Neumann 

algebra acting on /C. Since Z is second countable. Theorem 4, Ch II § 3.3 of [11] implies 
that 



M' = J Mjd\{z), 

MnM'= [ MzDMjdXiz). 
Jz 



Further, both M and M' are type I von Neumann algebras. Indeed, for almost every 
z E Z, Hz is a. group of type I, hence A^ is a representation of type I, that is, is a 
type I von Neumann algebra. Corollary 2, Ch II § 3.5 of [TT] implies that M is of type 
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I , again because Z is second countable. Finally, M' is of type I by Theorem 1 Ch. 1 
§ 8 of dl]. 

By applying twice (A50) of [12] we infer that there exists a countable family {P*}jg/ 
of non-zero pairwise orthogonal projections in MflM' with sum the identity such that 
both the reduced algebray M* := Mpi and the reduced algebra (M*)' = (M')p« are 
homogeneous. Since M and M' are decomposable, for each i G / the decomposition 
P* = PI dX{z) holds, where, for all z ^ Z, is a. projection in M^HM'^ and z P!. 
is a A-measurable field of operators. Proposition 6 Ch. II § 3.5 of [TT] implies that 

M' = [ Mid\{z) M'' = I Mi'dXiz) 
Jz Jz 

where, for all z & Z, Ml is the reduced algebra associated with P^. 
Furthermore, for almost every z & Z , the family {P^*}jg/ is pairwise orthogonal with 
sum the identity. Indeed, given i,j G / with i ^ j, Proposition 3 Ch. 2 § 2.3 in [TT] 
gives = PiPj = f P^P^dX^z), hence the Corollary of the cited section ensures that 
P^PJ = for almost all z. Since / is countable, then the above equality holds almost 
everywhere for all i,j G /. Given such a z, {P^^jjgj is a family of pairwise orthogonal 
projections, so that J2i ^1 converges to a projection P^ with respect to the strong 
operator topology, and so does P* converge to the identity. Proposition 4 Ch. 2 in 
[TTj § 2.3 and the uniqueness of the limit imply that P^ = id for almost every z. 

Fix i E I. Since M* and M*' are homogeneous, the very definition of homogeneous 
von Neumann algebra (see Ch. 3, § 3.1 in [TT]) and the canonical isomorphism given 
by Proposition 5 Ch. 1, § 2.4 in [TT], give 

M' = ® Cidc™. 

where is a maximal abelian algebra acting on a suitable closed subspace T* C /C. 
Denote by Q the orthogonal projection onto Cei (g)C™' ®T* G M*, where ei is the first 
element of the canonical basis of any C^. The corresponding reduced algebra of M* 
is id^d, ® A\ Furthermore, if Q is the orthogonal projection onto Cei (g> T* G (Mq)', 
the corresponding reduced algebra of (Mq)' is A^. Hence, reasoning as before, T* is a 
direct integral of a A-measurable field 2; i— )■ 7^' of Hilbert subspaces of P^/C^ and A" is 
a decomposable algebra, so that 

A'= [ A'.dXiz) 
Jz 

where, for almost every z, A\ is a maximal von Neumann algebra on T.^. Proposition 3 
Ch. 2 § 3.4 in [H] gives 

M' = I £(C'^0 ® Cidcd, ®AidX{z), 
Jz 

"^The algebra of operators obtained by restricting to the subspace Wi = PiK and then projecting 
back to Wi, hence a von Neumann algebra on Wi. 
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SO that (ii) of Proposition 1 Ch. 2 § 3.4 imphes, for almost every z, 

(57) Pile, = C^' ® C™' Mi = £(C*) ® C idc^, ® A^,. 

Hence, Proposition 3 Ch. Ill § 3.2 and Theorems 1 and 2 of Ch. 1 §. 7.3 give the 
existence of a unitary operator J* from P^/Cz onto ® C"^' (8> a;*) such that 

(58) JiMUr^ = CiC"') ® Cidc". ® L~(fi;, w^) 

where Ql is a locally compact second countable space and ul is a measure with support 

ni 

The previous arguments and the compactness assumption imply that there exists a 
negligible set C Z such that for all z & Z \ N, fl571) holds true for any z G / and 
is compact. Hence, with the notation used in fHSjl . for z ^ N we put 

n!, = card{s G i^z : dim/C^^s = (ij, = rrii}. 

Hence in ( 135]) the set can be chosen as {1, ... , n!,} if 1 < ra^ < +oo, N if ra!, = Kq, 
if = 0, and the measure cu* as the corresponding counting measure. By construction, 
2; I— )■ T^Ms a measurable field of Hilbert spaces and Proposition 1 Ch. II, § 1.4 of [TT] 
implies that for any cardinal p the set Z*^ = {z ^ Z\N : nl = p} is A-measurable and, 
for each i G / 

(59) [jz'P = Z\N. 

p 

Clearly for all z G Z^^ we have Ql = fi*^, where 

({l,...,p} l<p<+oo 
n'P := <N p = Ko 

[0 p = 0, 

so that the von Neumann algebra L°°(f2!,, w*) is equal to ^°°(f2*^), independently of z. 
Lemma 2 Ch. 2 § 3. of [II] implies that the unitary operator J] : C^^^C^'^C^ Pj/C^ 
can be chosen in such a way that z ^ Jl is A-measurable. The previous arguments 
show that the relevant indices n = {i,p, k) run on a countable set that will be denoted 
A/". Define m„ = m,- and 



Ji {C^^ ® C{ei} ® C{efc}) z G Z^p, 1 < k < p, p > Q 
{0} otherwise. 

Summarizing, we finally obtain the following facts, which entail the result. 

i) For each n G A^, the map z h-> /C" is A-measurable since z ^ JI is A-measurable 
field of operators. 

ii) For almost aX\ z & Z and for each n ^ M the Hilbert space /C" is invariant with 
respect to A^, the corresponding restriction is irreducible and the restriction to 
J:(C'^' (8)C™' ® C{efc}}) is a factor representation, see fl58l) . Thus 

Ji (C^^ ® C™' ® C{efc}) =1Q® C^- A|;cj«5c™n = A|,C5 ® id . 

In particular, for each j = 1, . . . ,m„ the field z J*(C'^' ® C{ej} Cjcfc}) is 
A-measurable. 
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iii) For almost all z ^ for each n ^ n' the restriction of A^; to /C" and A^"' are 
inequivalent, provided that both spaces are different from zero (l58ll . 

iv) For almost all z ^ Z, {ICl ® C^^j^g^ is a family of pairwise orthogonal closed 
subspace with sum /C^, by (1591) and the definition of /C". 

□ 

By means of the intertwining operator S given by Theorem 0, the direct decompo- 
sition flMl) gives rise to a corresponding decomposition of the mock-metaplectic repre- 
sentation U . Hence, the abstract theory of [IB] applies and one can characterize the 
admissible vectors for U . However, we can apply directly Corollary HJ We need a last 
technical lemma concerning the measurability of the map z i— >■ wo\{Hz) (compare with 
Lemma 18 of UHl). 



Lemma 7. Assume that for almost every ^ Y the stabilizer Hy^ is compact and 
define 

vo\{Hy^) = ds 
where ds is the unique Haar measure of Hy^ such that 

^{h)a{h~^)dh = / / ^{h{y)s)ds] dT^(^y^){y) G C,(F). 



H 

Then: 

(i) for all yo and h G H, Yo\{Hhiyo]) = Ag(/i"^) vo\{Hy^); 
(a) the map yo t— t- vol(ifj^,)) is Lebesgue measurable. 
Furthermore, given a Borel measurable section o : Z ^ Y , the map 

dim/C^ 

is X-measurable; if G is unimodular, it is independent of the choice of a. 

Proof. Fix a continuous / G L^{Y) such that f{y) > for all y &Y. The definition of 
Tz (see Theorem H]) and (iii) of Theorem [11] imply that / is r^-integrable for A-almost 
every z E Z. Clearly, the function {yo,h) H- f{h[yQ])a{h~^) is continuous on y x if. 
Given yQ G Y, let z = vr(|/o). Hence we can choose yo as the origin of n'^^z) and 
define ds as the unique Haar measure of Hy^ = for which (15^ holds true. By (ii) of 
Theorem [m for almost all yo eY, 



0< / f{h[yo])a{h ) dh = / f{hys[yo]) ds ] dT^(^y,^){y) 



H 




(60) = V0l(i/yj J^fiy)dT^{yo){y) < +00 

since hys[yo] = y; the first inequality is due to the fact / > and the last follows from 
/ G L^{Y). Clearly yo h-> f{h[yo])a{h~^) dh is Lebesgue-measurable as well as yo H- 
Jy f\y)^'^-^{yo){y) is Lebesgue measurable and strictly positive, so that yo i-> \o\{Hy^^) is 
A-measurable, too. The fact that the map z h- )■ dim/C^ is A-measurable for all n G A/" 
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is a consequence of Proposition 1, Ch. 2 § 1.4 of 

If yi = £[yo] for some i E H, whence 7r(?/o) = T^ivi), then by ([60 

vol{HyJ [ f{y)dr^iy,M= [ f{h[yi])a{h~')dh 
Jy J h 



h ^ hr^ ) = AH{r^)a{£) / f{h[yo])a{h~^)dh 
= AG{r')vo\iHy,) l^fiy)dT^^y,)iy). 



The second half of the lemma is clear. □ 

We are ready to state our main result on the admissible vectors of G. We distinguish 
according as to weather G is unimodular or not. We consider first the unimodular case, 
compare with Eq. (4.14) of Theorem 4.22 in [TSj . 

Theorem 8. Assume that G is unimodular and that for almost every z E Z the 
stabilizer Hz is compact. The representation U is reproducing if and only if the following 
two conditions hold true: 
(i) the integral 

(61) I ""•''*:'''''^»dA(.) 



J z vol Hz 

is finite; 

(a) for all n E M and for almost every z E Z for which /C" ^ 
(62) m„ < dim?{^ 

where the notation is as in (l53i) and (15^ . 
Under the above equivalent conditions, rj is an admissible vector for U if and only if 



dim ]Cz,n 



EE 

n£Af j=l ^ 

where {z i— >■ £^j}j>i is any measurable field of Hilbert bases for z ^ "H". 

Proof. We use the same notation as in Remark [HI Theorem [H] and Corollary H] with 
^dhy) = 1 give that 77 G L'^{X) is an admissible vector for U if and only if F = Wrj G 
H satisfies the condition that follows. Given n G J\f, for almost every z E Z for which 
/C" 7^ {0} (see Remarks [7] and [8]) , for alH, j = 1, . . . ,m„ 

that is, the family {F!^-}^^ is orthogonal in "H" and normalized with square norm equal 
to dim }Cz,n/vol Hz- 

As a consequence, if 77 is an admissible vector, then clearly fl62|) holds true and, by f l56|) . 
we have that 

dim/C^,„\ f card$~^(?/o) ,w x 
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and (l6Ti) follows. Conversely, define F G "H such that, for all j = 1, ... , and i > I 



/ dim /C^,„ 

4,(z) = 5,,^^^^ a.e.zGZ, 

which is possible due to (!62|) . All the functions /j^^ are A- measurable by Lemma [3 
Finally, fl6T]) and the last string of equalities imply < +oo. □ 

We now consider the non-unimodular case. For all n G A/" and for almost every z E Z 
we define the positive self-adjoint injective operator d^^n acting on "H" by multiplication 
as in f H9|) . namely 

(4,nF.,n)(^?) = ^^l^^cig) F.M g e G. 

Theorem 9. Assume that G is non-unimodular and that the stabilizer is compact 

for almost every z E Z . Then U is reproducing and rj E L'^{X) is an admissible vector 

for U if and only if Srj = J2neAf XljlTi / dX{z) (g) cj is such that 

z 

(i) for all n E M and i = 1, . . . ,m„, the map z t— )■ Fz^n,i is a measurable field of 

vectors for {H"}; 
(a) for alln E M and for almost all z E Z for which /C" ^ 

{d-^'F:,^ = ^^^ 2, J = 1, . . . , m„ 

(^) f \\F:,rn,dX{z) <+oo. 

neJV j=l ■'^ 

Proof. The fact that ?7 is admissible if and only if (i), (ii) and (iii) hold true is similar to 
the proof of Theorem [HI The non-trivial part is the existence of an admissible vector. 
This fact is a consequence of Theorem 4.23 of [18], whose proof can be repeated in our 
setting. We report the main ideas. 

Fix a strictly positive sequence such that XlngA^ Xll^'i '^n,* < +oo. For almost every 
z E Z the stability subgroup i/^ is compact, hence the modular function defines a 
continuous surjective : vr"^(z) — (0, +oo) by t^z{v) = '^G{h{y)), where h{y)[o{z)] = 
y. Therefore there exists a subset Yz^n,i of Ti'^iz) with strictly positive r^-measure such 
that for all y E Yz^n,i 

X / N / «n,i vol Hz 

sup A^(?/) < — — . 

y&,.„,^ climyL^^„ 

By Lemma [7] we may select a family of A- measurable fields {z i— )■ -F"^}™!^ of vectors in 

\l'2 1/2 

dom ciz,n , that are orthonormal with respect to the scalar product induced by dz,n 
with the property that the support with respect to of the map y t-)- \\F'z^j{h{y))\\'^^ ^ 
is contained in Yz^n,i- Thus, (iii) is satisfied because 

IIF" ||2 < suD dim/C,,„AG(/i(y)) 
" ^'^"^"-.JoL vol/7. 

Finally, (i) and (ii) are true by construction. □ 
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4. Examples 



We now discuss the examples introduced in Section 2. 

4.1. Example [T]. Here the map $ is the identity so that the set of critical points 
reduces to the empty set and Assumption 1 is satisfied with the choice X = Y = 'R'^ 
(recall that n = d) and a{h)P{h) = 1 for all h & H. Assumption 2 is the fact that 
the semi-direct product M'' x if is regular. In general, nothing more specific can be 
said on the parameter space Z and the measure A on it, other than what was said in 
the comments following Assumption 2. Clearly, for all y G M'^, $~^(|/) is a singleton, 
the corresponding measure Vy is trivial, so that Theorem [3] states that r} G L'^{X) is 
admissible for [/, for A-almost z E Z ii and only if 



where uq is a fixed origin in tt~^{z). Since the above equation holds true for any other 
point in 7r~^(z), it follows that r/ is a weak admissible vector in the sense of Definition 7 
of [in]. Theorem 6 of the cited paper proves that Assumption 2 is essentially necessary 
to have weak admissible vectors, (see the comment at the end of Section [23]). Corol- 
lary [2] guarantees that the stabilizers are compact for almost every z E Z. Hence 
the results of Section [X7I hold true. Clearly, for almost every /C^ = C, A/^ is a singleton 
and rrtn = dim(/C") = 1, so that U is always reproducing if G is non-unimodular. 
Otherwise, it is such if and only if J^{vo\ Hz)~^dX{z) is finite, which is precisely the 
content of Theorem 19 of [TH]. See also Section 5 of [IB]. The presence of volHz is due 
to a different normalization of the Haar measures on the stabilizers. 

4.2. Example [21. In this example n = 2 and d = 1 so that U is not reproducing. This 
fact is well known since G has a non-compact center and U is irreducible. 

4.3. Example [3]. The main result here is about groups of the form ([TTl) with n = d, 
namely: 

Theorem 10. Let n = d. If the H -orbits o/$(R'^) are locally closed, the restriction of 
the metaplectic representation to G is reproducing if and only if G is non-unimodular 
and Hy is compact for almost every y G $(M"'). 

In order to prove Theorem [TOj which could be stated under the slightly more general 
hypothesis that $ is a homogeneous polynomial without referring to the symplectic 
group, we need an auxiliary result which is of some interest by itself and whose main 
idea goes back to [27] . 

Proposition 5. Let n < d. Assume that ^ is a homogeneous map of degree p > 
and that the action on M°' is linear. If U is a reproducing representation, then G is 
non-unimodular. 

The proof is based on the following lemma. 

Lemma 8. Let n < d. Assume that ^ is a homogeneous map of degree p > and that 
the action on M*^ is linear. If rj is an admissible vector for U , then for any 5 G M+, the 
dilated vector V S'^P~'^r]^ is also admissible. 
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Proof. Put q = np — d. The assumption of $ implies that for all x G Mf^, a G M" and 
6 G M+, 

(63) m6x),6-Pa) = mx),a). 

Clearly, \/5^rj^ G L'^{X) and, for all / G L^(X), the linearity of x h-)- /i.a; gives 



G 




/(x)/3(/i)-5x 



X e 



27ri($(xO,a), 



r/(/i-\((5-'x))dx 



'^dadh 
a{h) 



G 



\fiSx)\'dx 

2 II ^l|2 



(reproducing formula) = 5^+^^^ 



so that V (5''?7 is an admissible vector for U. 

Proof of Proposition O By contradiction, assume that G is unimodular. Fix 6 G 
Choose an admissible vector t] G L^(X). Then 



X 



\r]{x)\^dx = 6"^ 
(reproducing formula for rj ) = 5~'^ 
(a I— !■ —a, /i I—)- /i~"^) 
{a^{h^)-M) 



\r]\x)\^dx 



X 




\{ri\U,H^)\ 



H J A 



,dadh 
a{h) 



6-' 
5-" 




H J A 




H J A 



, c , .oOiih) dadh 

ma,H)V^r^)\'^G{hy^ 



{q = np-d) = r ""-^ / I (r/, UgV6^v')\'dg 

Jg 

(reproducing formula for y/S^rj) = 6~^^ / \ri{x)\'^dx = 6 

Jx 

Since ||?7|| ^ and np ^ 0, this is a contradiction. 



□ 



□ 



of Theorem\T^ Clearly Assumption 2 is satisfied. Suppose that U is reproducing. 
Since $ is quadratic. Proposition |5] implies that G is non-unimodular and Theorem [1] 
gives that the set C of critical points is negligible. The Jacobian criterion implies 
that for all y G $(7?.) the fiber ^~^{y) fl M is finite (see Appendix iBl). Theorem [6] 
implies that for almost all y G $(7^) equality fH2l) holds true and, as a consequence 
of (i) of Corollary O the corresponding stabilizer Hy is compact. Conversely, if G is 
non-unimodular and almost every stabilizer is compact, the set of critical points is a 
proper Zariski closed subset of R'^, so that it is negligible. Theorem IH] implies that U 
is reproducing. □ 
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Theorem |9] characterizes the admissible vectors. However, one can also apply directly 
Theorem [3l taking into account that is a finite set. 

Corollary 4. A function r] G L'^{X) is an admissible vector for U if and only if for A- 
almost every z G Z , there exists y G 7i^^{z) such that for all points Xi, . . . Xm G 

r/(/i~^Xi)r/(/i-i.Xj) ^^^^^^^^ = (J$)(xi) 5ij i,j = 1,...M. 

If the above equation is satisfied for a pair Xi,Xj G $~^(?/), then it holds true for any 
pair s.Xi, s.Xj G with s G Hy. 

Proof. We apply Theorem [31 Given z & Z and y G ^'^{z) for which fl25p holds true, 
formula (1751) gives that 



M r. 

Or 



Arguing as in the proof of Proposition HI f l25l) is equivalent to 

J^vih~\xi)ri{h-\xj) ^^^^^^^^^ = i.m){x,)Sij I, J = l,...Ny. 

The last claim is clear because Hy is compact so that for all s & we have a{s) = 
f3{s) = 1 and hence the equality 

{M){h.x) = {J<^){x)a{hy^/3{hy^ heH. 

□ 

As an example, we apply the above corollary to the metaplectic representation re- 
stricted to the shearlet group G = TDS{2). Notice that 

( J$)(xi, X2) = xl/2 a{i, t) = t^+^ t) = t-^. 

We set X = {{xi,X2) G | a^i 7^ 0}, which is an if-invariant open set with full 
Lebesgue measure and Y = $(X) = ]R_ x M, is a transitive free if-space. We choose as 
origin the point yo = (—1/2,0) so that $^^(1/0) = {(iljO)}. Since for any h = {£,t) G 
H 

h~\{i,o) = {t^,t''-k), 

a function 77 G L^{X) is an admissible vector if and only if 

/ , / 1 1 „N dtdt 1 

(64) / \vit--,t'--^)\'T^ = 7; 

J{0,+oo)xR ^ 

/ , / 1 - 1 „N ,o dtdt 1 

(65) / |ry(-t.,-t7-5£)|2 = 



(66) / r/(t^r-^£)r/(-ti, -t^-|£) ^ = 0. 

^(0,+oo)xR t 

To recover the usual admissibility condition, put X± = {(xi,X2) : ±xi > 0} and define 
the unitary operator R± : L'^{Y) — )■ L^(X±) 

{R±f){xuX2) = /($(a;i,a;2))|J$(xi,a;2)|5, 
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SO that 

which clarifies the connection with the shearlet representation, see [20]. Denote by 
fy± = R^ri\x±i equations (|M|) and ( 1^ become 

1 1 f^"^ 1 
fl±{ — t, — tU)? dtdi=-. 

With the change of variables uji = —\t and uj2 = —^f^i, whose Jacobian is ^f^, they 
become 



duiduj2 



\V±[^U^2)\ 2 



Similarly, f l66|) becomes 



■ duJiduj2 



T)+{uJi,uj2)r]-{uJi,oj2) 2 — = 0- 

One should compare this with formula (2.1) in [23] • Note that U is equivalent to 
two copies of the irreducible representation Ind^2(x); where x is the character of 
(ai,a2) (-> e™i. 

4.4. Example IH With the choice X = \ |o} and Y = = (0, +oo) Assump- 

tion 1 is satisfied because X is an //-invariant open set whose complement has zero 
Lebesgue measure. The group H acts freely on Y so that Assumption 2 holds true 
and Z reduces to a singleton. We choose yo = 1 as the origin of the orbit, whose sta- 
bilizer is the compact group Hi = T. Since G is non-unimodular, U is reproducing by 
Theorem O In order to characterize its admissible vectors note that in Theorem H] the 
relatively invariant measure on F is ri = dy. Furthermore, the map ^ t-?- (cos sin 
is diffeomorphism of onto the Riemannian submanifold $~^(1) = {x1 + x\ = 1}. 
The Riemannian measure on is d^ so that, for all ip G Cc{X) 

di 



/ Lp{xi,X2)dui{xi,X2) = I v5(cos^,sin^)' 
Jx Jo 



2 



Put h{y) = {y/y, 0) so that h{y)[l] = y. Then (139|) says that the Haar measure on T is 
dO/Air because 

SO that vol T = i. 

The representation Ai of T on L'^{X, ui) ^ L?{S^^ dC,/2) is the regular representation, 
and 



L2(X,z/i)^0C{e-«} 
Am-©' 



le 

where each component is irreducible and any two of them are inequivalent. 
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Since any g = {a,t,6) can be written as g = (0, t, 0)(t^a, 0, 6'), any function F G "H 
can be identified with its restriction to M+ due to (K2). Further, (K3) becomes 

/•oo /*oo 

/ \Fi^)\'y-'dy = / |F(t)|22rMt<+oo. 
Jo Jo 

Hence we have the following unitary identifications 

n - L'^{R+,2t-^dt,L'^{S\d^/2)) ~ x S^r^dtd^). 

The unitary map S : L'^{X) — > "H is given explicitly by 

(^/) (t , = t ,,-1 ) (0 = t/ (t cos t sin . 

For n G Z, the space Tin carrying the representation induced by e~^'^*''~*"^ is 

Hn = {F e L2(M+ X T.r^dtdi) I = Fn{t)e'''^ , Fn G L^(M+, r^rft)}. 

If 7] G iv2(X), then Srj = Enez^^^'"^ ^i^^ F„ G L2(M+, t-irft). It follows that rj is an 
admissible vector if and only if, for any n E 1^, 

>d^\ _2 J dim Kn 
volT 

since dim/C„ = 1. By the change of variable t = ^/y, this is equivalent to 

1 



\F^{t)\H-^dt 

71 



Finally, since 



Fn{t) = ^ [ tr/(tcose,tsinOe-'"«rfe =: tfi{t,n), 

^TT Jo 

the set of admissible vectors consists of the Lebesgue measurable functions 77 : — C 
such that 

r+oo 

/ \fi{t,n)\Hdt < +00 7/GL2(M^) 

r+oo 

/ \fi{t,n)\'^r^dt = - Vn G Z. 

Jo 

4.5. Example [5l In this example Assumption 1 is satisfied with the choice X = 
\ {^2 = 0} and Y = = M \ {0}, because X is a if-invariant open set whose 

complement has zero Lebesgue measure. The group H acts freely on Y so that As- 
sumption 2 holds true and Z reduces to a singleton. We choose yo = 1 as the origin of 
the orbit so that the corresponding stabilizer is the non-compact group Hi = M*. To 
prove that G a reproducing group, we use Theorem [61 In Theorem |4] the relatively in- 
variant measure on Y is ti = dy. Furthermore, the map ^ H- (^,1) is a diffeomorphism 
of M onto the Riemannian submanifold <I>~^(1) = {x2 = 1}. The Riemannian measure 
on R is d^ and (J$)(x) = 1, so that ([75]) gives for all (f G Cc{X) 



(p{xi,X2)diyi{xi,X2) = / (p{^,l)d^. 



X 
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Put h{y) = {y,0) so that h{y)[l] = y. Then (139|) says that the Haar measure on M is 
db because 

[ ip{t,b)\t\^db = [ { [ ip{y,b)db]dy. 
The representation Ai of M on L'^{X, ui) ^ dC,) is the regular representation, and 

L^{X,ui) ^ f Cduj 
Jr 

Ai,b~ [ e-^^'^^^duj 
Jr 

where each component is irreducible, any two of them are inequivalent and the inter- 
twining operator is given by the Fourier transform. 

Since any g = (a, t,b) E G can be written as g = (0, t, 0){ta, 0, b), any function F eT-L 
can be identified with its restriction to M* due to (K2) and we have the following unitary 
identifications 

H ~ L'^{R\t-^dt,L'^{^~\l),ui)) ~ L\M.'^,y-^dydO- 
The unitary map 5* : L'^{X) — L'^{M.'^,y~^dyd$,) is given explicitly by 

{Sf){y.O = \t\'-f{y,0- 

Theorem implies that r] G L'^{X) is an admissible vector if and only if for all u G 
L^{R,dO 

[\um'd^= [ ( [\{u,\y\"^Ai,,{Sv){y,-Wdb) \y\-'dy 
Jr Jr \Jr J 

= [ ( [\uico)my,co)\'dco) \y\-'dy 
Jr \Jr J 

where we use that A{h{y)) = a{h{y))~^ = \y \ and where" denotes the Fourier transform 
with respect to C,- It follows that the set of admissible vectors is the set of Lebesgue 
measurable functions 77 : — t- C such that 

[ ( [ \fj{y,u)\'du] dy < +00 ^ r]e L^M^) 
Jr \Jr / 

\fi{y,uj)\'^\y\~^dy = 1 for almost every a; G M. 

This set is clearly non empty: take for example any strictly positive continuous function 
a e (M) and define 
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Appendix A. Appendix: some measure theory revisited 

In this Appendix we review some known facts that are somehow hard to locate in 
the hterature in a way that is both easily accessible and stated under the assumptions 
that we are making. The spaces X and Y are as in Section E] and are regarded as 
measure spaces with respect to the Lebesgue measure, denoted dx and dy respectively. 

A.l. Disintegration of measures. We start by adapting to our setting some facts 
from integration theory on general locally compact spaces. The main reference for the 
issues at hand is [3]. Hereafter, Cc{X) denotes the space of compactly supported contin- 
uous functions on X, endowed with the usual locally convex (separable) inductive limit 
topology, for which a sequence (</?n)ngN in C'c(A) converges to zero if there exists a com- 
pact set K such that supp v^n C K for all n and lim„^ooSup3,g;^|(^„(x)| = 0. We denote 
by M[X) the topological dual of Cc(A); when equipped with the a{M{X),Cc{X))- 
topology, the topological dual of M(X) is again Cc{,X) ([SU], Th. IV.20). Since X 
is second countable, the Riesz-Markov representation theorem uniquely identifies the 
measures with the positive elements of M{X). By the word measure on a locally com- 
pact second countable topological space, we mean a positive measure defined on the 
Borel (T-algebra, which is finite on compact subsets. 

The following theorem, in some sense a version of Fubini's theorem, summarizes the 
main properties of the kind of disintegration of measures we are concerned with. The 
main point here, though, is the possibility of extending the disintegration from Cc to 

. We state it for X and F, but it also holds verbatim if we replace X and Y with 
two arbitrary locally compact second countable topological spaces. 

Theorem 11. Suppose that uj is a measure on X and p a measure on Y and let 
: X ^ Y be a u-measurable map. Assume further that {ojy} is a family of measures 
on X such that 

(a) Uy is concentrated on '$~^{y) for all y G Y; 

(b) J ip{x)d(jj{x) = J (p{x)duy{x)^ dp{y) for all ip G Cc{X). 

Then, for any u-measurable function / : X — C the following facts hold true: 
(i) / is ujy -measurable for almost every y G Y ; 



(ii) / is u-integrable if and only if J \f{x)\duy{x) j dp{y) is finite; 

(iii) if f is u-integrable, then f is Uy-integrable for p-almost every y &Y , the func- 
tion (defined almost everywhere) y H- f{x)duy{x) is p-integrable, and 

(67) ^ f{x)du{x) = f{x)duy{x)^ dp{y); 

(iv) if {u'y} is another family of measures on X satisfying (a) and (b), then u'y = Uy 
for p-almost all y eY . 

Proof. The theorem is essentially contained in [3] , scattered in several statements. For 
the proof of (i), (ii) and (iii) we quote from Chapter 5, and for the proof of (iv) from 
Chapter 6. 
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Statement (i) is the content of a) Prop. 4, § 3.2, taking into account that, since 
it is second countable, X is a-compact and, a fortiori, w-moderated (a subset is u- 
moderated if it is contained into the union of a countable sequence of compact subsets 
and a w- negligible set). 

As for (ii), since X is second countable. Prop. 2, § 3.1, guarantees that the family 
ip{x)duy{x) is p-adequate in the sense of Def. 1, § 3.1. The equivalence of the two 
conditions in (ii) is then the content of the Corollary at the end of § 3.2. 

As for (iii), it is just Th. 1, § 3.3, observing that any function is w-moderated since 
X is cu-moderated (a function is cu-moderated if it is null on the complement of a 
a;-moderated subset). 

Finally, for (iv), by assumption JyUydp{y) = Jyu'ydpd/), where the integral is a 
scalar integral of vector valued functions taking values in M(X). Now Lemma 1, § 3.1 
ensures that Cc{X) has a countable subset which is dens^ in Cc{X) with respect to 
the a{Cc{X), M{X)) topology, so that, by Remark 2 in §1.1, it is enough to show that 
for any ip G Cc{X) and for p-almost every y G F 

(p{x)dujy{x) = / (p{x)duj'y{x). 

X J X 

This is in turn equivalent to proving that 

(68) ^ {^j^ v{x)dujy{x)^ i{y)dp{y) = ^{x)du'y{x)^ i{y)dp{y) 

holds for all ip G Ca{X) and ^ G Cc{Y). Fix then ip G Cc{X) and ^ G Cc{Y), and put 
f{x) = C,{'^{x))ip{x). This function is w-measurable since \l/ is w-measurable and ^ and 
ip are continuous, it is bounded since both ^ and ip are bounded, and it has a compact 
support since ip is compactly supported. Hence / is u- integrable. Applying twice fl67j) 
we get 



(69) \^jj{^{x)Mx)doOy{x)j dp{y) = \^jj{^{x)Mx)dij'y{x)j dp{y). 
Given y eY, (a) implies that ,^(\I^(x)) = C,{y) for w^^-almost all a; G X, so that 

^{'^{x))ip{x)duy{x)^ dp{y) = j (^j f{x)duy{x)^ ^iy)dp{y), 

and similarly for the right hand side of (l69i) . Hence (l68i) is true and the claim is 
proved. □ 

The integral formula (b) will be written for short 

(70) ^ / ^y^p^y^' 



^ It is proved there that there exists a countable subset S C Cc{X) such that for every ip G Cc{X) 
there is a sequence (</'n)neN in S converging to (p uniformly and \ip„\ < \ipo\- 
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A. 2. Direct integrals. Next we recall the definition of direct integral, following [T7] . 
Hereafter we assume that the hypotheses of Theorem [11] are satisfied. Fix a countable 
family {ipk}keN dense in Cc{X), and hence also in every L'^{X,Uy), with y eY. The 
map y H- {fk,fi)uiy is p-measurable since it is p-integrable by hypothesis (b) of Theo- 
rem [TT] Under these circumstances, {fk}keN is called a p-measurable structure for the 
family of Hilbert spaces {L"^ {X , Uy)} . The direct integral fy L'^{X,Uy)dy is defined as 
the set consisting of all the families {fy} satisfying: 

(Dl) fyeL^{X,Uy) for aWyeY; 



(D2) JJfy\\t^dpiy)<+oo; 

(D3) y {fy, v?fc)wy is p-measurable for all k eN. 



Two families J-" = {fy} and Q = {gy} are identified if for almost every y ^ Y fy = Qy 
as elements in L'^{X,ujy). The space Jy L'^{X,Uy)dp{y) is a Hilbert space under 



{J^,Q) = / {fy.9y)u.ydp{y). 



Y 



Since Cc{X) has a dense countable subset, see Footnote (D3) is equivalent to 

(D3') y (-7- {fy,(p)uiy is p-measurable for all ip G Cc{X), 

so that, as long as we choose the functions of {(pk}k£N in Cc{X), the measurable 
structure is independent of the choice of the particular family. 

Proposition 6. Given f G L'^{X,uj), there exists a unique family {fy} in the Hilbert 
space direct integral fy L'^{X,Uy)dp{y) such that, for almost every y eY , the equality 
fy{x) = f{x) holds for Uy-almost every x E X . Furthermore, the map f i— )■ {fy} is a 
unitary operator from L"^ {X , u) onto jy L'^{X,ujy)dp{y) . 

Proof. By hypothesis (b) of Theorem [TTl for every (f G Cc{X) we have 



(p{x)duj{x) = J yj ^{x)duyj dpiy). 

Given a functioi]!^ / : X — ?■ C which is square-integrable with respect to hence in 
particular cu-measurable, (i) of Theorem [TT] implies that / is Wj^-measurable for almost 
every y eY . Further, since |/P is integrable with respect to cu, (iii) of the same theorem 
ensures that |/p is cjy-integrable for almost all y G F, the map y i— )■ f-^\f{x)\'^dujy{x) 
is integrable, and 



(71) JJf{x)\'dco{x) = ^JJf{x)\'dcOy{x)j dpiv). 

Hence there is a p-full set Y' G Y such that, if y G Y', f is square-integrable with 
respect to Uy. For y E Y' define fy to be the equivalence class of / in L'^{X,Uy) and, 
for y ^ Y', put fy = 0. 



Here it is important that / is a function, and not an equivalence class modulo a.e. equality. 
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We claim that T = {fy} is in Jy L'^{X,ujy)dp{y). By fITT]) . conditions (Dl) and (D2) 
are clearly satisfied. To prove (D3'), take ip G Cc{X). Clearly, fTp is w-integrable and 
hence, by (iii) of Theorem [TTl it is w^-integrable for almost every y and 

y^ f{x)ip{x)dUJy{x) = {fy,^)oJy 

J X 

is integrable, hence measurable. Therefore / i— )■ J-" is a well defined map from the space 
of square-integrable functions on X to fy L'^{X,Uy)dp{y), it is linear and, by fl7T]l . 

(72) / \f{x)\'du;{x)= I \\fy\\l/p{y). 

J X Jy 

Hence, it defines an isometry from w) into Jy L'^{X,Uy)dp{y) and, by construc- 

tion, for almost every y E Y, the equality fy{x) = f{x) holds for cj^-almost every 
xeX. 

We claim that the isometry / i— )■ J-" is surjective. It is enough to prove that for any 
family J-" whose members fy are positive, there exists a positive / G L'^{X, u) such that, 
for almost every y E Y, the equality fy{x) = f{x) holds for w^-almost every x G X. 
Take then such an J-". First of all, we show that the family of measures {fy ■ Uy} is 
scalarly integrable with respect to p. This is equivalent to saying that for all ip G Cc{X) 
the function y i— >■ F^{y) = J^ip{x)fy{x)duy{x), certainly well defined because (Dl) 
implies that ipfy is cuy-integrable for every y E Y, is p-integrable. Indeed, (D3') says 
that is p-measurable, whereas Holder's inequality and Cauchy-Schwartz give 

JjF^iy)\dpiy) < JjiplUMfllMv) 

so that by (D2) and (1721) applied to ip yield 

j^\F^{y)\dp{y) < C\M < +00. 

Hence the claim is proved and p = Jy{fy ■ ujy)dp{y) defines a measure. We show next 
that /X is a measure with bas^l^;. This will produce the required / that maps to J^. The 
Lebesgue-Nikodym theorem (see Th. 2 ,§ 5.5, Ch. 5 of [5]) ensures that it is enough to 
prove that any compact subset K <Z X ioi which uj{K) = satisfies p{K) = 0. Take 
such a K. Item (iii) of Theorem [TT] applied to the characteristic function xk gives that 
for almost every y eY, K is cuy-negligible and, a fortiori, fy ■ Wy- negligible. Thus, fl67j) 
with (jj = p, Uy = f ■ Uy and / = xk yields 

p{K) = (^jjy{x)dujy{x)^ dp{y) = 0. 

Hence there exists a locally integrable positive function / such that f-u = p. Moreover, 
if (p E Cc{X), ipf is integrable, so that again (iii) of Theorem [TT] tells us that, for almost 

measure which is the product ^p ■ C of a measure £ by a locally >C-integrable positive function 
ip is called a measure with base £ (see Def. 2, § 5.2, Ch. V in [3]). 
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every y eY, ipf is Wy-integrable, the map y t-)- j^ip{x)f{x) duy{x) is integrable and 
by definition of /i 

Lp{x)fy{x) dujy{x) ) dp{y) = / ip{x) dp{x) 

Y \JX J J X 

(^j ^{x)f{x)duy{x)^dp{y). 

By tlie above equality, (iv) of Theorem [TT] may be apphed to infer that for almost every 
y E Y the equality f = fy holds a;j,-almost everywhere. Finally, (D2) gives 

Hence (iii) of Theorem ITT] implies that / is square integrable. The equivalence class of 
/ in w) is then the element required to prove surjectivity. □ 

Both L^(X, cj) and each of the spaces L'^{X,Uy) can be identified with subspaces of 
M{X) simply by viewing their elements as continuous linear functionals on Cc{X) via 
integration with respect to u and Uy, respectively. Further, (iv) of Theorem [TT] implies 
that saying that for almost every y eY the equality fy{x) = f{x) holds for w^^-almost 
every x E X is equivalent to 



/ ■ ^ = / ify- ^y)dp{y), 



Y 



in the sense that the map Y — )■ M(X), y fy ■ Uy is p-scalarly-integrable. These 
remarks together with Proposition E] imply that 

(73) L\X,u) = j^L\X,u;y)dp{y) 
by means of the equality in M{X) 

(74) f = J^fydp{y), 
where the integral is a scalar integral. 

A. 3. The coarea formula for submersions. Below we give a simple proof of the 
Coarea Formula for submersions; the general case is due to Federer (TB] . Suppose that 
n < d and let X C M'^ be an open set. Recall that a C^-map $ : X — )■ is called a 
submersion if its differential is surjective for all x E X . For every y eY = $(X), 
let dv'^{x) denote the volume element of the Riemannian submanifold $~^(?/) and by 
J$ the Jacobian. We introduce the measure Vy on X by 

(75) ^y{E) = [ T^^T^' E e B{X). 



{y)nE (J'^Kx)' 

It is worth observing that Uy is finite on compact sets and concentrated on $~^(?/). 
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Theorem 12 (Coarea formula for submersions). Suppose that $ : X — j- zs a 
submersion. Then 



(76) dx = J dvy dy, 

where dx and dy are the Lebesgue measures on M'^ and M", respectively. 
Proof. We must show that 

f{x)dx= f ( [ f{x)-^^]dy 



X JY \J X 



{J^){x] 



holds for every / G Cc{X). Fix xq G X. Since ^^xo is surjective, the Inverse Mapping 
Theorem implies (Corollary 5.8 in [2B]) that there exists a diffeomorphism ^> : U xV ^ 
W such that 

(77) ^{^{z,y))=y z e U, y e V, 

where U is an open subset of M'^^", V is an open subset of M" and W is an open 
neighborhood of xq- 

Take / G Cc{X). For any such /, since supp / is compact, by choosing a suitable 
finite covering if necessary, we can always assume that supp f G W. The change of 
variables formula and Fubini's Theorem give 



(78) / f{x)dx= / / f{^{z,y)){J^){z,y)dz\dy. 

Jw Jv \Ju J 

To obtain the coarea formula we simply compute the Jacobian . Observe that for 
any given y = ^'(■,?/) is a diffeomorphism from U onto W fl $~^(|/), regarded 

as a submanifold. In particular, using this local chart, the volume element at the point 

X = '^{^z, y) is given by 



(79) dyyix) = ^/d^tfi^^)Zm^]dz. 

Taking the derivatives of fl77|) with respect to z and y separately, we obtain 

(80) ^*'f{z,y) Di'^(^z,y) = 0, '^**(2,?/) D2'^(z,y) = Inxn- 

Fix {z,y) G UxV and let Pi denote the orthogonal projection from M"^ onto ker 

and P2 = I — Pi the orthogonal projection onto [ker $^,^(2 y)]-*-, which is a subspace of 

dimension n because $ is a submersion. From fISUl) it follows that 

(81) P2(/^l*)(.,,) = 0, P2(I)2*)(.,,) = ($**(.,,) O 

where l : [ker $^.,5,(2 j^)]-*- — )■ M'^ is the natural injection. Let R G 0{d) be the rotation 
that takes ker $*^(2,jy) onto the 2;-hyperplane (first d — n coordinates) and its orthogonal 
complement onto the |/-hyperplane (last n coordinates), so that RPi{z,y) = z and 
RP2{z,y) = y. Then (IHTj) imply 



A B 
C 
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where A = R{Di<i/)(^,^y), B = RPi{D2^){z,y) and C = i?P2(A*)(^,y). Therefore 



(J*)(^,y) = \ det m,^,,y)\ = I det All det C| =- V^et [*(^^)*.(^^)« 



'det [^*^(z,y)^^*^(z,y)] 

where we have used flHT]) . Taking f l79|) into account, for x = we have 

which inserted in (ITHI) yields the result. □ 

Appendix B. The Jacobian criterion 

We show below that Theorem 16.19 in [15] implies that for all y G $(7^) the fiber 
$^^(?/) is finite. First of all, we can view $ as a polynomial map from into itself, 
so we write $ = {fi, . . . , fd). Without loss of generality we assume further that y = 
0. Following [15], we write S = C[Xi, . . . , X^] and we denote by / the ideal in S 
generated by /i, . . . , /d. We are interested in its radical a//, which decomposes as an 
intersection, unique up to order, of prime ideals a/7 = Pi fl ■ ■ ■ fl P^- Hence V{I) = 
{w G : fi{w) = ■ ■ ■ = fd{w) = 0} = Vi U ■ ■ ■ U Vs, the corresponding decomposition 
into irreducible components, namely Vi = Z{Pi). Under the present circumstances, 
dimVi = d — codim(Pj), where the latter is the KruU codimension of Pj. Clearly, 
codim(Pj) = d if and only if Vi is a singleton. Suppose that dim Vj > for some j. We 
will show that at the points w ^ Vj the Jacobian determinant 

,M(w) = det ( ^(w' 

vanishes. Suppose by contradiction that J$(w;) 7^ 0. Now, the codimension of Ip. in 
Sp. is equal to codim(Pj) because Pj is a minimal prime of /. By assumption, this is 
strictly smaller than d. By the Jacobian criterion, the Jacobian matrix taken modulo 
Pj has rank strictly less than d. This means that J$ G Pj. But w G l^, which implies 
that J$(w) = 0, a contradiction. Therefore $~^(0) ft 7^ does not intersect irreducible 
components with positive dimension, hence it is a finite set. 
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